DETAILED ACTION
This action is in response to Applicant’s Amendment ("Response”) received on January 29, 2026 in response to the Office Action dated November 10, 2025. This action is made Final.
Claims 1-3, 5-10, 12-16, and 18-28 are pending.
Claims 1, 8, 14, 21, and 25 are independent claims.
Claims 1-3, 5-10, 12-16, and 18-28 are rejected.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Applicant’s Response
In Applicant’s Response, Applicant amended claims 1, 8, 14, and 21-25, added claims 26-28, cancelled claims 4, 11, and 17, and submitted arguments against the prior art in the Office Action dated November 10, 2025.
Based on Applicant’s amendments and remarks, the Examiner withdraws the rejection of claims under 35 USC 101.
Based on Applicant’s amendments and remarks, the Examiner withdraws the objection to claim 24.
Based on Applicant’s amendments and remarks, the Examiner withdraws the rejection of claims 22 and 23 under 35 USC 112.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 3, 5, 7, 25, 26, and 28 are rejected under 35 U.S.C. 103 as being unpatentable over Wu, L. et al. “Scalable Zero-shot Entity Linking with Dense Entity Retrieval.” Available at https://arxiv.org/abs/1911.03814 on 29 September 2020 (hereafter, WU), in view of the applicant’s admitted prior art, and further in view of Hermans, A. et al. “In Defense of the Triplet Loss for Person Re-Identification.” Available at https://arxiv.org/abs/1703.07737 on 21 November 2017 (hereafter, HERMANS)
Claim 1:
Regarding claim 1, WU teaches “A method, comprising: extracting, with a processor, a mention… from a free-form text”:
([Abstract] “This paper introduces a conceptually simple, scalable, and highly effective BERT-based entity linking model (a machine learning model that is run by a processor), along with an extensive evaluation of its accuracy-speed trade-off. We present a two-stage zero-shot linking algorithm, where each entity is defined only by a short textual description. The first stage does retrieval in a dense space defined by a bi-encoder that independently embeds the mention context and the entity descriptions (This explicitly mentions “retrieval” of a mention). Each candidate is then re-ranked with a cross-encoder, that concatenates the mention and entity text. Experiments demonstrate that this approach is state of the art on recent zero-shot benchmarks (6 point absolute gains) and also on more established non-zero-shot evaluations (e.g. TACKBP-2010), despite its relative simplicity (e.g. no explicit entity embeddings or manually engineered mention tables). We also show that bi-encoder linking is very fast with nearest neighbour search (e.g. linking with 5.9 million candidates in 2 milliseconds), and that much of the accuracy gain from the more expensive cross-encoder can be transferred to the bi-encoder via knowledge distillation.”)
And further:
([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
The above figure clearly shows the “mention” of “Jaguar” being extracted from a “free form text” (“My kids really enjoyed a ride in the Jaguar!”))
Further, WU teaches “encoding the mention with an embedding space encoder, … wherein the encoding creates an encoded representation of the mention in a multi-dimensional embedding space” -
([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
The above figure clearly shows the “mention” of “Jaguar” being extracted from a “free form text” (“My kids really enjoyed a ride in the Jaguar!”). Further, we see the “Bi-Encoder” and the “Cross-Encoder”, both embedding space encoders, creating an encoded representation of the mention (“Jaguar”) in a multi-dimensional embedding space (dense space). The “Dense Space” is used to embed the encoded representations in at least two dimensions, as shown in the figure, making it a multi-dimensional embedding space.)
Further, WU teaches “wherein the embedding space encoder implements a machine learning model trained using contrastive learning”:
([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
Contrastive learning, as defined in the art, is a machine learning method that learns by comparing pairs of data to pull similar examples together and push dissimilar ones apart in a representation space, without relying on explicit labels. As shown in the above figure, the extracted mention “Jaguar” is tested to see if it more closely relates to a junior roller coaster or the luxury vehicle brand, by pulling the similar examples together in the “dense space” (a representation space) and pushing the dissimilar ones apart, such as “Ferrari”. It is shown calculating a 0.8 score for the junior roller coaster and a 0.2 score for the luxury vehicle brand, and thus, mapping it as the junior roller coaster, using this method as opposed to labelled data. In doing so, figure 1 illustrates the exact process of contrastive learning.)
Further, WU teaches “mapping, with the processor, the encoded representation of the mention to an encoded representation of an entity in the multi-dimensional embedding space, wherein the entity is extracted from a knowledge base…”:
([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
As mentioned above, we see the process of contrastive learning result in it mapping the encoded representation of the mention (“Jaguar”) to an entity (“a junior roller coaster”) in the multi-dimensional embedding space (“dense space”). The “Dense Space” is used to embed the encoded representations in at least two dimensions, as shown in the figure, making it a multi-dimensional embedding space.
And further:
([3. Definition and Task Formulation, Entity Linking] “Given an input text document D = {w1, …, wr} and a list of entity mentions MD = {m1, …, mn}, the output of an entity linking model is a list of mention-entity pairs {(mi, ei)}
PNG
media_image2.png
17
45
media_image2.png
Greyscale
where each entity is an entry in a knowledge base (KB) (e.g. Wikipedia),
PNG
media_image3.png
23
46
media_image3.png
Greyscale
. We assume that the title and description of the entities are available, which is a common setting in entity linking (Ganea and Hofmann, 2017; Logeswaran et al., 2019). We also assume each mention has a valid gold entity in the KB, which is usually referred as in-KB evaluation. We leave the out-of-KB prediction (i.e. nil prediction) to future work.”)
Further, WU teaches “outputting the entity whose encoded representation maps to the encoded representation of the mention”:
([3. Definition and Task Formulation, Entity Linking] “Given an input text document D = {w1, …, wr} and a list of entity mentions MD = {m1, …, mn}, the output of an entity linking model is a list of mention-entity pairs {(mi, ei)}
PNG
media_image2.png
17
45
media_image2.png
Greyscale
where each entity is an entry in a knowledge base (KB) (e.g. Wikipedia),
PNG
media_image3.png
23
46
media_image3.png
Greyscale
(Here we see the output being explicitly defined as a list of “mention-entity pairs” where each entity is an entry in a knowledge base. This correlates to the outputting the entity whose representation maps to the mention). We assume that the title and description of the entities are available, which is a common setting in entity linking (Ganea and Hofmann, 2017; Logeswaran et al., 2019). We also assume each mention has a valid gold entity in the KB, which is usually referred as in-KB evaluation. We leave the out-of-KB prediction (i.e. nil prediction) to future work.”)
WU fails to explicitly teach the mention being “…a mention of a computer application component…”, the free-form text being “…a textual description of a technology stack of the computer application…”, and the knowledge base being “…a knowledge base of computer components…”
However, as the specification of the present application admits at [0004-0005], “Application modernization” is a known field for “updating of a legacy application”, that “typically” (This word is used verbatim, showing it is well-known) involves obtaining information from “the legacy application’s technology stack”, which is “usually” (This word is used verbatim, showing it is well-known) in the form of “free-form textual descriptions.” Further, it admits the existence of these “mentions” of computer application components in the “technology stack” of the computer application, and the known aspect of “entity standardization”, explicitly stating “The same application component may be mentioned in different ways in different technology stacks. This poses a significant challenge to entity standardization, which aims to map various technical mentions to entities in a knowledge base. The knowledge base entities are words, phrases, acronyms, symbols, or the like, that are widely recognized by working professionals as referring to specific application components, (including computer components). Entities drawn from a knowledge base are thus standard references to application components. A mention extracted from a textual description, however, may take form – name, acronym, symbols, or the like – that differs from the name, acronym, or symbols that are widely recognized by computer professionals. This poses a significant challenge to mapping the mention to the correct entity, which is the aim of standardization.”
Given the specification’s admitted prior art and the disclosure of WU, it would be obvious to one of ordinary skill in the art to combine the methods taught in WU with the known field of use described in the present specification, prior to the effective filing date of the claimed invention, and that doing so would achieve the claimed invention.
One of ordinary skill in the art would be motivated to do so because, as the present specification points out at [0004], “The automatic extraction of such information from a textual description of a technology stack is a non-trivial task”, and as WU points out in its Abstract, “We also show that bi-encoder linking is very fast with nearest neighbour search (e.g. linking with 5.9 million candidates in 2 milliseconds), and that much of the accuracy gain from the more expensive cross-encoder can be transferred to the bi-encoder via knowledge distillation.”
Further, WU in view of the applicant’s admitted prior art fails to explicitly teach “wherein the embedding space encoder is trained using a hybrid batch-all and batch-hard online triplet loss mining”. However, analogous art, HERMANS, does teach this:
([2. Learning Metric Embeddings, the Triplet Loss, and the Importance of Mining] “The goal of metric embedding learning is to learn a function… which maps semantically similar points from the data manifold… onto metrically close points…
… A major caveat of the triplet loss, though, is that as the dataset gets larger, the possible number of triplets grows cubically, rendering a long enough training impractical. To make matters worse, f0 relatively quickly learns to correctly map most trivial triplets, rendering a large fraction of all triplets uninformative. Thus mining hard triplets becomes crucial for learning…
In a classical implementation, once a certain set of B triplets has been chosen, their images are stacked into a batch of size 3B… Given the fact that there are up to 6B2 - 4B possible combinations of these 3B images that are valid triplets, using only B of them seems wasteful. With this realization, we propose an organizational modification to the classic way of using the triplet loss: the core idea is to form batches by randomly sampling P classes (person identities), and then randomly sampling K images of each class (person)… Now, for each sample a in the batch, we can select the hardest positive and the hardest negative samples within the batch when forming the triplets for computing the loss, which we call Batch Hard…
…This new formulation of sampling a batch immediately suggests another alternative, that is to simply use all possible PK(PK - K)(K - 1) combinations of triplets, which corresponds to the strategy chosen in [9] and which we call Batch All”) This citation shows that HERMANS simultaneously uses both Batch-Hard and Batch-All online triplet loss mining.
It would be obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to combine the base reference of WU in view of the admitted prior art with the teachings of HERMANS because both references use triplet-loss mining.
One of ordinary skill in the art would be motivated to do so because, as pointed out by HERMANS’ conclusion, “We propose a variant (of triplet loss) that no longer requires offline hard negative mining at almost no additional cost. Combined with a pretrained network, we set the new state-of-the-art on three of the major ReID datasets.”
Claim 3:
Regarding claim 3, WU in view of the applicant’s admitted prior art teaches the limitations of claim 1. Further, WU teaches “wherein a backbone of the machine learning model implemented by the embedding space encoder is a context-sensitive, per-trained transformer language model”:
([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
As shown in the figure, a Bi-Encoder and Cross-Encoder are both used, which are both examples of deep pre-trained bidirectional transformers used in context-sensitive language modeling, as is also supported in Section 2, Related Work “…Humeau et al. (2019) studied different architectures to use deep pre-trained bidirectional transformers and performed detailed comparison of three different architectures, namely bi-encoder, poly-encoder, cross-encoder on tasks of sentence selection in dialogues. Inspired by their work, we use similar architectures to the problem of entity linking, and in addition, demonstrate that bi-encoder can be a strong model for retrieval. Instead of using the poly-encoder as a trade-off between cross-encoder and bi-encoder, we propose to train a bi-encoder model with knowledge distillation (Buciluundefined et al., 2006; Hinton et al., 2015) from a cross-encoder model to further improve the bi-encoder’s performances.”
Claim 5:
Regarding claim 5, WU in view of the applicant’s admitted prior art teaches the limitations of claim 1. Further, WU teaches “wherein the mapping is performed by determining a vectorial distance between the encoded representation of the mention and the encoded representation of the entity in the knowledge base”:
([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
Contrastive learning, as defined in the art, is a machine learning method that learns by comparing pairs of data to pull similar examples together and push dissimilar ones apart in a representation space, without relying on explicit labels. As shown in the above figure, the extracted mention “Jaguar” is tested to see if it more closely relates to a junior roller coaster or the luxury vehicle brand, by pulling the similar examples together in the “dense space” (a representation space) and pushing the dissimilar ones apart, such as “Ferrari”. In doing so, the distance between the encoded representation of the mention and the entity in the knowledge base is instrumental to the mapping.)
And further:
([4. Methodology] “Figure 1 shows our overall approach. The bi-encoder uses two independent BERT transformers to encode model context/mention and entity into dense vectors, and each entity candidate is scored as the dot product of these vectors. The candidates retrieved by the bi-encoder are then passed to the cross-encoder for ranking. The cross-encoder encodes context/mention and entity in one transformer, and applies an additional linear layer to compute the final score for each pair.”) This explicitly shows that the distance of each is a vectorial distance.
Claim 7:
Regarding claim 7, WU in view of the applicant’s admitted prior art teaches the limitations of claim 1. Further, WU teaches “mining a plurality of online databases for textual descriptions… compiling the textual descriptions in the knowledge base”:
([5.1 Datasets, TACKBP-2010] “…There are 1,074 and 1,020 annotated mention/entity pairs derived from 1,453 and 2,231 original news and web documents on training and evaluation dataset, respectively.”)
Further, WU teaches “encoding entities extracted from the textual descriptions in the multi-dimensional embedding space”:
([4.1 Bi-encode. Entity Modeling] “The entity representation… is also composed of word-pieces of the entity title and description (for Wikipedia entities, we use the first ten sentences as description). The input to our entity model is: [CLS] title [ENT] description [SEP] where title, description are word-pieces tokens of entity title and description, and [ENT] is a special token to separate entity title and description representation.”) Here we see entities extracted from the textual descriptions encoded into the embedding space, the dense space, as cited from Figure 1. The “Dense Space” is used to embed the encoded representations in at least two dimensions, as shown in the figure, making it a multi-dimensional embedding space.
Further, in light of the applicant’s admitted prior art, it teaches the textual descriptions being “…of computer application components” at [0004-0005], “Application modernization” is a known field for “updating of a legacy application”, that “typically” (This word is used verbatim, showing it is well-known) involves obtaining information from “the legacy application’s technology stack”, which is “usually” (This word is used verbatim, showing it is well-known) in the form of “free-form textual descriptions.” Further, it admits the existence of these “mentions” of computer application components in the “technology stack” of the computer application, and the known aspect of “entity standardization”, explicitly stating “The same application component may be mentioned in different ways in different technology stacks. This poses a significant challenge to entity standardization, which aims to map various technical mentions to entities in a knowledge base. The knowledge base entities are words, phrases, acronyms, symbols, or the like, that are widely recognized by working professionals as referring to specific application components. Entities drawn from a knowledge base are thus standard references to application components. A mention extracted from a textual description, however, may take form – name, acronym, symbols, or the like – that differs from the name, acronym, or symbols that are widely recognized by computer professionals. This poses a significant challenge to mapping the mention to the correct entity, which is the aim of standardization.”
Claim 25:
Wu teaches or suggests a method, comprising:
generating, with a processor, a plurality of … examples to train a contrastive learning model, wherein an embedding space encoder implements a machine learning model trained using contrastive learning (see ([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
Contrastive learning, as defined in the art, is a machine learning method that learns by comparing pairs of data to pull similar examples together and push dissimilar ones apart in a representation space, without relying on explicit labels. As shown in the above figure, the extracted mention “Jaguar” is tested to see if it more closely relates to a junior roller coaster or the luxury vehicle brand, by pulling the similar examples together in the “dense space” (a representation space) and pushing the dissimilar ones apart, such as “Ferrari”. It is shown calculating a 0.8 score for the junior roller coaster and a 0.2 score for the luxury vehicle brand, and thus, mapping it as the junior roller coaster, using this method as opposed to labelled data. In doing so, figure 1 illustrates the exact process of contrastive learning; ([5.1 Datasets, TACKBP-2010] “…There are 1,074 and 1,020 annotated mention/entity pairs derived from 1,453 and 2,231 original news and web documents on training and evaluation dataset, respectively.”) And further: ([4.1 Bi-encode. Entity Modeling] “The entity representation… is also composed of word-pieces of the entity title and description (for Wikipedia entities, we use the first ten sentences as description). The input to our entity model is: [CLS] title [ENT] description [SEP] where title, description are word-pieces tokens of entity title and description, and [ENT] is a special token to separate entity title and description representation.”) Here we see entities extracted from the textual descriptions encoded into the embedding space.);
extracting a mention … from a free-form text (see ([Abstract] “This paper introduces a conceptually simple, scalable, and highly effective BERT-based entity linking model (a machine learning model that is run by a processor), along with an extensive evaluation of its accuracy-speed trade-off. We present a two-stage zero-shot linking algorithm, where each entity is defined only by a short textual description. The first stage does retrieval in a dense space defined by a bi-encoder that independently embeds the mention context and the entity descriptions (This explicitly mentions “retrieval” of a mention). Each candidate is then re-ranked with a cross-encoder, that concatenates the mention and entity text. Experiments demonstrate that this approach is state of the art on recent zero-shot benchmarks (6 point absolute gains) and also on more established non-zero-shot evaluations (e.g. TACKBP-2010), despite its relative simplicity (e.g. no explicit entity embeddings or manually engineered mention tables). We also show that bi-encoder linking is very fast with nearest neighbour search (e.g. linking with 5.9 million candidates in 2 milliseconds), and that much of the accuracy gain from the more expensive cross-encoder can be transferred to the bi-encoder via knowledge distillation.”)
And further:
([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
The above figure clearly shows the “mention” of “Jaguar” being extracted from a “free form text” (“My kids really enjoyed a ride in the Jaguar!”)).;
encoding the mention with the embedding space encoder (see ([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
The above figure clearly shows the “mention” of “Jaguar” being extracted from a “free form text” (“My kids really enjoyed a ride in the Jaguar!”). Further, we see the “Bi-Encoder” and the “Cross-Encoder”, both embedding space encoders, creating an encoded representation of the mention (“Jaguar”) in a multi-dimensional embedding space (dense space). The “Dense Space” is used to embed the encoded representations in at least two dimensions, as shown in the figure, making it a multi-dimensional embedding space.)
WU still fails to explicitly teach the mention being “… a mention of a computer application component … wherein the free-form text is a textual description of a technology stack of the computer application.”
However, as the specification of the present application admits at [0004-0005], “Application modernization” is a known field for “updating of a legacy application”, that “typically” (This word is used verbatim, showing it is well-known) involves obtaining information from “the legacy application’s technology stack”, which is “usually” (This word is used verbatim, showing it is well-known) in the form of “free-form textual descriptions.” Further, it admits the existence of these “mentions” of computer application components in the “technology stack” of the computer application, and the known aspect of “entity standardization”, explicitly stating “The same application component may be mentioned in different ways in different technology stacks. This poses a significant challenge to entity standardization, which aims to map various technical mentions to entities in a knowledge base. The knowledge base entities are words, phrases, acronyms, symbols, or the like, that are widely recognized by working professionals as referring to specific application components (including computer components). Entities drawn from a knowledge base are thus standard references to application components. A mention extracted from a textual description, however, may take form – name, acronym, symbols, or the like – that differs from the name, acronym, or symbols that are widely recognized by computer professionals. This poses a significant challenge to mapping the mention to the correct entity, which is the aim of standardization.”
Given the specification’s admitted prior art and the disclosure of WU, it would be obvious to one of ordinary skill in the art to combine the methods taught in WU with the known field of use described in the present specification, prior to the effective filing date of the claimed invention, and that doing so would achieve the claimed invention.
One of ordinary skill in the art would be motivated to do so because, as the present specification points out at [0004], “The automatic extraction of such information from a textual description of a technology stack is a non-trivial task”, and as WU points out in its Abstract, “We also show that bi-encoder linking is very fast with nearest neighbour search (e.g. linking with 5.9 million candidates in 2 milliseconds), and that much of the accuracy gain from the more expensive cross-encoder can be transferred to the bi-encoder via knowledge distillation.”
Further, Wu fails to explicitly disclose “… triplet examples … iteratively adjusting, by the processor, parameters of the contrastive learning model during an initial sequence of training epochs using batch-all mining with the triplet examples; and further adjusting, by the processor, parameters of the contrastive learning model during a subsequent sequence of training epochs using batch-hard mining as the parameters converge to a final set of values”.
However, analogous art, HERMANS does teach “…triplet examples…”:
([2. Learning Metric Embeddings, the Triplet Loss, and the Importance of Mining] “The goal of metric embedding learning is to learn a function… which maps semantically similar points from the data manifold… onto metrically close points…
… A major caveat of the triplet loss, though, is that as the dataset gets larger, the possible number of triplets grows cubically, rendering a long enough training impractical. To make matters worse, f0 relatively quickly learns to correctly map most trivial triplets, rendering a large fraction of all triplets uninformative. Thus mining hard triplets becomes crucial for learning…”)
Further, HERMANS teaches “iteratively adjusting, by the processor, parameters of the… model during an initial sequence of training epochs using batch-all mining with the triplet examples”
([2. Learning Metric Embeddings, the Triplet Loss, and the Importance of Mining] “The goal of metric embedding learning is to learn a function… which maps semantically similar points from the data manifold… onto metrically close points…
… A major caveat of the triplet loss, though, is that as the dataset gets larger, the possible number of triplets grows cubically, rendering a long enough training impractical. To make matters worse, f0 (a function adjusting the parameters) relatively quickly learns to correctly map most trivial triplets, rendering a large fraction of all triplets uninformative. Thus mining hard triplets becomes crucial for learning…
In a classical implementation, once a certain set of B triplets has been chosen, their images are stacked into a batch of size 3B… Given the fact that there are up to 6B2 - 4B possible combinations of these 3B images that are valid triplets, using only B of them seems wasteful. With this realization, we propose an organizational modification to the classic way of using the triplet loss: the core idea is to form batches by randomly sampling P classes (person identities), and then randomly sampling K images of each class (person)… Now, for each sample a in the batch, we can select the hardest positive and the hardest negative samples within the batch when forming the triplets for computing the loss, which we call Batch Hard…
…This new formulation of sampling a batch immediately suggests another alternative, that is to simply use all possible PK(PK - K)(K - 1) combinations of triplets, which corresponds to the strategy chosen in [9] and which we call Batch All”) This citation shows that HERMANS simultaneously uses both Batch-Hard and Batch-All online triplet loss mining, and continuously adjusting for each sample in the batch, until a final set of values is converged.
Further, HERMANS teaches “further adjusting, by the processor, parameters of the… model during a subsequent sequence of training epochs using batch-hard mining as the parameters converge to a final set of values”
([2. Learning Metric Embeddings, the Triplet Loss, and the Importance of Mining] “The goal of metric embedding learning is to learn a function… which maps semantically similar points from the data manifold… onto metrically close points…
… A major caveat of the triplet loss, though, is that as the dataset gets larger, the possible number of triplets grows cubically, rendering a long enough training impractical. To make matters worse, f0 (a function adjusting the parameters) relatively quickly learns to correctly map most trivial triplets, rendering a large fraction of all triplets uninformative. Thus mining hard triplets becomes crucial for learning…
In a classical implementation, once a certain set of B triplets has been chosen, their images are stacked into a batch of size 3B… Given the fact that there are up to 6B2 - 4B possible combinations of these 3B images that are valid triplets, using only B of them seems wasteful. With this realization, we propose an organizational modification to the classic way of using the triplet loss: the core idea is to form batches by randomly sampling P classes (person identities), and then randomly sampling K images of each class (person)… Now, for each sample a in the batch, we can select the hardest positive and the hardest negative samples within the batch when forming the triplets for computing the loss, which we call Batch Hard…
…This new formulation of sampling a batch immediately suggests another alternative, that is to simply use all possible PK(PK - K)(K - 1) combinations of triplets, which corresponds to the strategy chosen in [9] and which we call Batch All”) This citation shows that HERMANS simultaneously uses both Batch-Hard and Batch-All online triplet loss mining, and continuously adjusting for each sample in the batch, until a final set of values is converged.
It would be obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to combine the base reference of WU with the teachings of HERMANS because both references use triplet-loss mining.
One of ordinary skill in the art would be motivated to do so because, as pointed out by HERMANS’ conclusion, “We propose a variant (of triplet loss) that no longer requires offline hard negative mining at almost no additional cost. Combined with a pretrained network, we set the new state-of-the-art on three of the major ReID datasets.”
Claim 26:
Regarding claim 26, WU in view of the applicant’s admitted prior art in view Hermans teaches the limitations of claim 25. Further, WU teaches “mining a plurality of online databases for textual descriptions… compiling the textual descriptions in the knowledge base”:
([5.1 Datasets, TACKBP-2010] “…There are 1,074 and 1,020 annotated mention/entity pairs derived from 1,453 and 2,231 original news and web documents on training and evaluation dataset, respectively.”)
Further, WU teaches “encoding entities extracted from the textual descriptions in the multi-dimensional embedding space”:
([4.1 Bi-encode. Entity Modeling] “The entity representation… is also composed of word-pieces of the entity title and description (for Wikipedia entities, we use the first ten sentences as description). The input to our entity model is: [CLS] title [ENT] description [SEP] where title, description are word-pieces tokens of entity title and description, and [ENT] is a special token to separate entity title and description representation.”) Here we see entities extracted from the textual descriptions encoded into the embedding space, the dense space, as cited from Figure 1. The “Dense Space” is used to embed the encoded representations in at least two dimensions, as shown in the figure, making it a multi-dimensional embedding space.
Further, in light of the applicant’s admitted prior art, it teaches the textual descriptions being “…of computer application components” at [0004-0005], “Application modernization” is a known field for “updating of a legacy application”, that “typically” (This word is used verbatim, showing it is well-known) involves obtaining information from “the legacy application’s technology stack”, which is “usually” (This word is used verbatim, showing it is well-known) in the form of “free-form textual descriptions.” Further, it admits the existence of these “mentions” of computer application components in the “technology stack” of the computer application, and the known aspect of “entity standardization”, explicitly stating “The same application component may be mentioned in different ways in different technology stacks. This poses a significant challenge to entity standardization, which aims to map various technical mentions to entities in a knowledge base. The knowledge base entities are words, phrases, acronyms, symbols, or the like, that are widely recognized by working professionals as referring to specific application components. Entities drawn from a knowledge base are thus standard references to application components. A mention extracted from a textual description, however, may take form – name, acronym, symbols, or the like – that differs from the name, acronym, or symbols that are widely recognized by computer professionals. This poses a significant challenge to mapping the mention to the correct entity, which is the aim of standardization.”
Claim 28:
Regarding claim 28, Wu in view of applicant’s admitted prior art in view of Hermans teaches or suggests the limitations of claim 25. Wu further teaches or suggests “wherein a backbone of the machine learning model implemented by the embedding space encoder is a context-sensitive, per-trained transformer language model”:
([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
As shown in the figure, a Bi-Encoder and Cross-Encoder are both used, which are both examples of deep pre-trained bidirectional transformers used in context-sensitive language modeling, as is also supported in Section 2, Related Work “…Humeau et al. (2019) studied different architectures to use deep pre-trained bidirectional transformers and performed detailed comparison of three different architectures, namely bi-encoder, poly-encoder, cross-encoder on tasks of sentence selection in dialogues. Inspired by their work, we use similar architectures to the problem of entity linking, and in addition, demonstrate that bi-encoder can be a strong model for retrieval. Instead of using the poly-encoder as a trade-off between cross-encoder and bi-encoder, we propose to train a bi-encoder model with knowledge distillation (Buciluundefined et al., 2006; Hinton et al., 2015) from a cross-encoder model to further improve the bi-encoder’s performances.”
Claims 2, 6, and 27 are rejected under 35 U.S.C. 103 as being unpatentable over WU in view of the applicant’s admitted prior art, in view of Hermans, as applied to claims above, and further in view of Reimers, N. et al. “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.” Available at https://arxiv.org/abs/1908.10084 on 27 August 2019 (hereafter, REIMERS).
Claim 2:
Regarding claim 2, WU in view of the applicant’s admitted prior art, in view of Hermans, teaches the limitations of claim 1. Further, WU in view of the applicant’s admitted prior art fails to explicitly teach “wherein the machine learning model implemented by the embedding space encoder is a Siamese neural network”. However, analogous art, REIMERS, does teach this:
([Abstract, paragraphs 2-3] “…In this publication, we present Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT. We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning tasks, where it outperforms other state-of-the-art sentence embeddings methods.”)
It would be obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to combine the base reference of WU in view of the applicant’s admitted prior art with the teachings of REIMERS because both references involve extracting components of free form text and matching them with knowledge in a database.
One of ordinary skill in the art would be motivated to do so because, as pointed out in REIMERS’ abstract above, “This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT” and “…it outperforms other state-of-the-art sentence embeddings methods.”
Claim 6:
Regarding claim 6, WU in view of the applicant’s admitted prior art teaches the limitations of claim 5. Further, WU in view of the applicant’s admitted prior art fails to explicitly teach “wherein the vectorial distance is determined based on a cosine similarity between the encoded representation of the mention and the encoded representation of the entity in the knowledge base”. However, analogous art, REIMERS, does teach this:
([Page 1, Introduction, Col. 2, Last paragraph] “…we developed SBERT. The siamese network architecture enables that fixed-sized vectors for input sentences can be derived. Using a similarity measure like cosine-similarity or Manhatten / Euclidean distance, semantically similar sentences can be found. These similarity measures can be performed extremely efficient on modern hardware, allowing SBERT to be used for semantic similarity search as well as for clustering.”)
It would be obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to combine the base reference of WU in view of the applicant’s admitted prior art with the teachings of REIMERS because both references involve extracting components of free form text and matching them with knowledge in a database.
One of ordinary skill in the art would be motivated to do so because, as pointed out in REIMERS’ abstract above, “This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT” and “…it outperforms other state-of-the-art sentence embeddings methods.”
Claim 27:
Regarding claim 27, WU in view of the applicant’s admitted prior art in view of Hermans teaches the limitations of claim 1. Further, WU in view of the applicant’s admitted prior art fails to explicitly teach “wherein the machine learning model implemented by the embedding space encoder is a Siamese neural network”. However, analogous art, REIMERS, does teach this:
([Abstract, paragraphs 2-3] “…In this publication, we present Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT. We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning tasks, where it outperforms other state-of-the-art sentence embeddings methods.”)
It would be obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to combine the base reference of WU in view of the applicant’s admitted prior art with the teachings of REIMERS because both references involve extracting components of free form text and matching them with knowledge in a database.
One of ordinary skill in the art would be motivated to do so because, as pointed out in REIMERS’ abstract above, “This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT” and “…it outperforms other state-of-the-art sentence embeddings methods.”
Claims 8, 10, 12, 14, 16, 18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over WU in view of Singh, H. “Everything you Need to Know About Hardware Requirements for Machine Learning.” Available at https://www.einfochips.com/blog/everything-you-need-to-know-about-hardware-requirements-for-machine-learning/ on 24 February 2019 (hereafter, SINGH), in view of the applicant’s admitted prior art, and further in view of Hermans.
Claim 8:
Regarding claim 8, WU teaches “extracting a mention… from a free-form text”:
([Abstract] “This paper introduces a conceptually simple, scalable, and highly effective BERT-based entity linking model (a machine learning model that is run by a processor), along with an extensive evaluation of its accuracy-speed trade-off. We present a two-stage zero-shot linking algorithm, where each entity is defined only by a short textual description. The first stage does retrieval in a dense space defined by a bi-encoder that independently embeds the mention context and the entity descriptions (This explicitly mentions “retrieval” of a mention). Each candidate is then re-ranked with a cross-encoder, that concatenates the mention and entity text. Experiments demonstrate that this approach is state of the art on recent zero-shot benchmarks (6 point absolute gains) and also on more established non-zero-shot evaluations (e.g. TACKBP-2010), despite its relative simplicity (e.g. no explicit entity embeddings or manually engineered mention tables). We also show that bi-encoder linking is very fast with nearest neighbour search (e.g. linking with 5.9 million candidates in 2 milliseconds), and that much of the accuracy gain from the more expensive cross-encoder can be transferred to the bi-encoder via knowledge distillation.”)
And further:
([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
The above figure clearly shows the “mention” of “Jaguar” being extracted from a “free form text” (“My kids really enjoyed a ride in the Jaguar!”))
Further, WU teaches “encoding the mention with an embedding space encoder … wherein the encoding creates an encoded representation of the mention in a multi-dimensional embedding space”:
([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
The above figure clearly shows the “mention” of “Jaguar” being extracted from a “free form text” (“My kids really enjoyed a ride in the Jaguar!”). Further, we see the “Bi-Encoder” and the “Cross-Encoder”, both embedding space encoders, creating an encoded representation of the mention (“Jaguar”) in a multi-dimensional embedding space (dense space). The “Dense Space” is used to embed the encoded representations in at least two dimensions, as shown in the figure, making it a multi-dimensional embedding space.)
Further, WU teaches “wherein the embedding space encoder implements a machine learning model trained using contrastive learning”:
([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
Contrastive learning, as defined in the art, is a machine learning method that learns by comparing pairs of data to pull similar examples together and push dissimilar ones apart in a representation space, without relying on explicit labels. As shown in the above figure, the extracted mention “Jaguar” is tested to see if it more closely relates to a junior roller coaster or the luxury vehicle brand, by pulling the similar examples together in the “dense space” (a representation space) and pushing the dissimilar ones apart, such as “Ferrari”. It is shown calculating a 0.8 score for the junior roller coaster and a 0.2 score for the luxury vehicle brand, and thus, mapping it as the junior roller coaster, using this method as opposed to labelled data. In doing so, figure 1 illustrates the exact process of contrastive learning.)
Further, WU teaches “mapping the encoded representation of the mention to an encoded representation of an entity in the multi-dimensional embedding space, wherein the entity is extracted from a knowledge base…”:
([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
As mentioned above, we see the process of contrastive learning result in it mapping the encoded representation of the mention (“Jaguar”) to an entity (“a junior roller coaster”) in the multi-dimensional embedding space (“dense space”) The “Dense Space” is used to embed the encoded representations in at least two dimensions, as shown in the figure, making it a multi-dimensional embedding space.
And further:
([3. Definition and Task Formulation, Entity Linking] “Given an input text document D = {w1, …, wr} and a list of entity mentions MD = {m1, …, mn}, the output of an entity linking model is a list of mention-entity pairs {(mi, ei)}
PNG
media_image2.png
17
45
media_image2.png
Greyscale
where each entity is an entry in a knowledge base (KB) (e.g. Wikipedia),
PNG
media_image3.png
23
46
media_image3.png
Greyscale
. We assume that the title and description of the entities are available, which is a common setting in entity linking (Ganea and Hofmann, 2017; Logeswaran et al., 2019). We also assume each mention has a valid gold entity in the KB, which is usually referred as in-KB evaluation. We leave the out-of-KB prediction (i.e. nil prediction) to future work.”)
Further, WU teaches “outputting the entity whose encoded representation maps to the encoded representation of the mention”:
([3. Definition and Task Formulation, Entity Linking] “Given an input text document D = {w1, …, wr} and a list of entity mentions MD = {m1, …, mn}, the output of an entity linking model is a list of mention-entity pairs {(mi, ei)}
PNG
media_image2.png
17
45
media_image2.png
Greyscale
where each entity is an entry in a knowledge base (KB) (e.g. Wikipedia),
PNG
media_image3.png
23
46
media_image3.png
Greyscale
(Here we see the output being explicitly defined as a list of “mention-entity pairs” where each entity is an entry in a knowledge base. This correlates to the outputting the entity whose representation maps to the mention). We assume that the title and description of the entities are available, which is a common setting in entity linking (Ganea and Hofmann, 2017; Logeswaran et al., 2019). We also assume each mention has a valid gold entity in the KB, which is usually referred as in-KB evaluation. We leave the out-of-KB prediction (i.e. nil prediction) to future work.”)
WU fails to explicitly teach “A system, comprising: One or more processors configured to initiate operations”, the mention being “…a mention of a computer application component…”, the free-form text being “…a textual description of a technology stack of the computer application…”, and the knowledge base being “…a knowledge base of computer components…”
However, analogous art, SINGH, does teach “A system, comprising: One or more processors configured to initiate operations”:
([Hardware requirements for machine learning] “The first thing you should determine is what kind of resource does your task requires. Let’s have a look how different tasks will have different hardware requirements:
If your tasks are small and can fit in a complex sequential processing, you don’t need a big system (but a system is needed). You could even skip the GPUs altogether. A CPU such as i7–7500U can train an average of ~115 examples/second. So, if you are planning to work on other ML areas or algorithms, a GPU is not necessary.”) This describes the need for a system with some type of processor, a GPU or CPU, in order to train and use machine learning models.
It would be obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to combine the base reference of WU with the teachings of SINGH because WU teaches the use of machine learning in is methods, while SINGH teaches the requirements for machine learning methods.
One of ordinary skill in the art would be motivated to do so because without meeting the requirements of any given software, the operations of that software will not function.
WU in view of SINGH still fails to explicitly teach the mention being “…a mention of a computer application component…”, the free-form text being “…a textual description of a technology stack of the computer application…”, and the knowledge base being “…a knowledge base of computer components…”
However, as the specification of the present application admits at [0004-0005], “Application modernization” is a known field for “updating of a legacy application”, that “typically” (This word is used verbatim, showing it is well-known) involves obtaining information from “the legacy application’s technology stack”, which is “usually” (This word is used verbatim, showing it is well-known) in the form of “free-form textual descriptions.” Further, it admits the existence of these “mentions” of computer application components in the “technology stack” of the computer application, and the known aspect of “entity standardization”, explicitly stating “The same application component may be mentioned in different ways in different technology stacks. This poses a significant challenge to entity standardization, which aims to map various technical mentions to entities in a knowledge base. The knowledge base entities are words, phrases, acronyms, symbols, or the like, that are widely recognized by working professionals as referring to specific application components. Entities drawn from a knowledge base are thus standard references to application components(including computer components). A mention extracted from a textual description, however, may take form – name, acronym, symbols, or the like – that differs from the name, acronym, or symbols that are widely recognized by computer professionals. This poses a significant challenge to mapping the mention to the correct entity, which is the aim of standardization.”
Given the specification’s admitted prior art and the disclosure of WU, it would be obvious to one of ordinary skill in the art to combine the methods taught in WU in view of SINGH with the known field of use described in the present specification, prior to the effective filing date of the claimed invention, and that doing so would achieve the claimed invention.
One of ordinary skill in the art would be motivated to do so because, as the present specification points out at [0004], “The automatic extraction of such information from a textual description of a technology stack is a non-trivial task”, and as WU points out in its Abstract, “We also show that bi-encoder linking is very fast with nearest neighbour search (e.g. linking with 5.9 million candidates in 2 milliseconds), and that much of the accuracy gain from the more expensive cross-encoder can be transferred to the bi-encoder via knowledge distillation.”
Further, WU in view of the applicant’s admitted prior art in view of Singh fails to explicitly teach “wherein the embedding space encoder is trained using a hybrid batch-all and batch-hard online triplet loss mining”. However, analogous art, HERMANS, does teach this:
([2. Learning Metric Embeddings, the Triplet Loss, and the Importance of Mining] “The goal of metric embedding learning is to learn a function… which maps semantically similar points from the data manifold… onto metrically close points…
… A major caveat of the triplet loss, though, is that as the dataset gets larger, the possible number of triplets grows cubically, rendering a long enough training impractical. To make matters worse, f0 relatively quickly learns to correctly map most trivial triplets, rendering a large fraction of all triplets uninformative. Thus mining hard triplets becomes crucial for learning…
In a classical implementation, once a certain set of B triplets has been chosen, their images are stacked into a batch of size 3B… Given the fact that there are up to 6B2 - 4B possible combinations of these 3B images that are valid triplets, using only B of them seems wasteful. With this realization, we propose an organizational modification to the classic way of using the triplet loss: the core idea is to form batches by randomly sampling P classes (person identities), and then randomly sampling K images of each class (person)… Now, for each sample a in the batch, we can select the hardest positive and the hardest negative samples within the batch when forming the triplets for computing the loss, which we call Batch Hard…
…This new formulation of sampling a batch immediately suggests another alternative, that is to simply use all possible PK(PK - K)(K - 1) combinations of triplets, which corresponds to the strategy chosen in [9] and which we call Batch All”) This citation shows that HERMANS simultaneously uses both Batch-Hard and Batch-All online triplet loss mining.
It would be obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to combine the base reference of WU in view of the admitted prior art with the teachings of HERMANS because both references use triplet-loss mining.
One of ordinary skill in the art would be motivated to do so because, as pointed out by HERMANS’ conclusion, “We propose a variant (of triplet loss) that no longer requires offline hard negative mining at almost no additional cost. Combined with a pretrained network, we set the new state-of-the-art on three of the major ReID datasets.”
Claim 10:
Regarding claim 10, WU in view of SINGH & the applicant’s admitted prior art and Hermans teaches the limitations of claim 8. Further, WU teaches “wherein a backbone of the machine learning model implemented by the embedding space encoder is a context-sensitive, per-trained transformer language model”:
([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
As shown in the figure, a Bi-Encoder and Cross-Encoder are both used, which are both examples of deep pre-trained bidirectional transformers used in context-sensitive language modeling, as is also supported in Section 2, Related Work “…Humeau et al. (2019) studied different architectures to use deep pre-trained bidirectional transformers and performed detailed comparison of three different architectures, namely bi-encoder, poly-encoder, cross-encoder on tasks of sentence selection in dialogues. Inspired by their work, we use similar architectures to the problem of entity linking, and in addition, demonstrate that bi-encoder can be a strong model for retrieval. Instead of using the poly-encoder as a trade-off between cross-encoder and bi-encoder, we propose to train a bi-encoder model with knowledge distillation (Buciluundefined et al., 2006; Hinton et al., 2015) from a cross-encoder model to further improve the bi-encoder’s performances.”
Claim 12:
Regarding claim 12, WU in view of SINGH & the applicant’s admitted prior art and Hermans teaches the limitations of claim 8. Further, WU teaches “wherein the mapping is performed by determining a vectorial distance between the encoded representation of the mention and the encoded representation of the entity in the knowledge base”:
([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
Contrastive learning, as defined in the art, is a machine learning method that learns by comparing pairs of data to pull similar examples together and push dissimilar ones apart in a representation space, without relying on explicit labels. As shown in the above figure, the extracted mention “Jaguar” is tested to see if it more closely relates to a junior roller coaster or the luxury vehicle brand, by pulling the similar examples together in the “dense space” (a representation space) and pushing the dissimilar ones apart, such as “Ferrari”. In doing so, the distance between the encoded representation of the mention and the entity in the knowledge base is instrumental to the mapping.)
And further:
([4. Methodology] “Figure 1 shows our overall approach. The bi-encoder uses two independent BERT transformers to encode model context/mention and entity into dense vectors, and each entity candidate is scored as the dot product of these vectors. The candidates retrieved by the bi-encoder are then passed to the cross-encoder for ranking. The cross-encoder encodes context/mention and entity in one transformer, and applies an additional linear layer to compute the final score for each pair.”) This explicitly shows that the distance of each is a vectorial distance.
Claim 14:
Regarding claim 14, WU teaches “extracting a mention… from a free-form text”:
([Abstract] “This paper introduces a conceptually simple, scalable, and highly effective BERT-based entity linking model (a machine learning model that is run by a processor), along with an extensive evaluation of its accuracy-speed trade-off. We present a two-stage zero-shot linking algorithm, where each entity is defined only by a short textual description. The first stage does retrieval in a dense space defined by a bi-encoder that independently embeds the mention context and the entity descriptions (This explicitly mentions “retrieval” of a mention). Each candidate is then re-ranked with a cross-encoder, that concatenates the mention and entity text. Experiments demonstrate that this approach is state of the art on recent zero-shot benchmarks (6 point absolute gains) and also on more established non-zero-shot evaluations (e.g. TACKBP-2010), despite its relative simplicity (e.g. no explicit entity embeddings or manually engineered mention tables). We also show that bi-encoder linking is very fast with nearest neighbour search (e.g. linking with 5.9 million candidates in 2 milliseconds), and that much of the accuracy gain from the more expensive cross-encoder can be transferred to the bi-encoder via knowledge distillation.”)
And further:
([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
The above figure clearly shows the “mention” of “Jaguar” being extracted from a “free form text” (“My kids really enjoyed a ride in the Jaguar!”))
Further, WU teaches “encoding the mention with an embedding space encoder, wherein the encoding creates an encoded representation of the mention in a multi-dimensional embedding space”:
([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
The above figure clearly shows the “mention” of “Jaguar” being extracted from a “free form text” (“My kids really enjoyed a ride in the Jaguar!”). Further, we see the “Bi-Encoder” and the “Cross-Encoder”, both embedding space encoders, creating an encoded representation of the mention (“Jaguar”) in a multi-dimensional embedding space (dense space). The “Dense Space” is used to embed the encoded representations in at least two dimensions, as shown in the figure, making it a multi-dimensional embedding space.)
Further, WU teaches “wherein the embedding space encoder implements a machine learning model trained using contrastive learning”:
([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
Contrastive learning, as defined in the art, is a machine learning method that learns by comparing pairs of data to pull similar examples together and push dissimilar ones apart in a representation space, without relying on explicit labels. As shown in the above figure, the extracted mention “Jaguar” is tested to see if it more closely relates to a junior roller coaster or the luxury vehicle brand, by pulling the similar examples together in the “dense space” (a representation space) and pushing the dissimilar ones apart, such as “Ferrari”. It is shown calculating a 0.8 score for the junior roller coaster and a 0.2 score for the luxury vehicle brand, and thus, mapping it as the junior roller coaster, using this method as opposed to labelled data. In doing so, figure 1 illustrates the exact process of contrastive learning.)
Further, WU teaches “mapping the encoded representation of the mention to an encoded representation of an entity in the multi-dimensional embedding space, wherein the entity is extracted from a knowledge base…”:
([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
As mentioned above, we see the process of contrastive learning result in it mapping the encoded representation of the mention (“Jaguar”) to an entity (“a junior roller coaster”) in the multi-dimensional embedding space (“dense space”) The “Dense Space” is used to embed the encoded representations in at least two dimensions, as shown in the figure, making it a multi-dimensional embedding space.).
And further:
([3. Definition and Task Formulation, Entity Linking] “Given an input text document D = {w1, …, wr} and a list of entity mentions MD = {m1, …, mn}, the output of an entity linking model is a list of mention-entity pairs {(mi, ei)}
PNG
media_image2.png
17
45
media_image2.png
Greyscale
where each entity is an entry in a knowledge base (KB) (e.g. Wikipedia),
PNG
media_image3.png
23
46
media_image3.png
Greyscale
. We assume that the title and description of the entities are available, which is a common setting in entity linking (Ganea and Hofmann, 2017; Logeswaran et al., 2019). We also assume each mention has a valid gold entity in the KB, which is usually referred as in-KB evaluation. We leave the out-of-KB prediction (i.e. nil prediction) to future work.”)
Further, WU teaches “outputting the entity whose encoded representation maps to the encoded representation of the mention”:
([3. Definition and Task Formulation, Entity Linking] “Given an input text document D = {w1, …, wr} and a list of entity mentions MD = {m1, …, mn}, the output of an entity linking model is a list of mention-entity pairs {(mi, ei)}
PNG
media_image2.png
17
45
media_image2.png
Greyscale
where each entity is an entry in a knowledge base (KB) (e.g. Wikipedia),
PNG
media_image3.png
23
46
media_image3.png
Greyscale
(Here we see the output being explicitly defined as a list of “mention-entity pairs” where each entity is an entry in a knowledge base. This correlates to the outputting the entity whose representation maps to the mention). We assume that the title and description of the entities are available, which is a common setting in entity linking (Ganea and Hofmann, 2017; Logeswaran et al., 2019). We also assume each mention has a valid gold entity in the KB, which is usually referred as in-KB evaluation. We leave the out-of-KB prediction (i.e. nil prediction) to future work.”)
WU fails to explicitly teach “A computer program product, the computer program product comprising: one or more computer-readable storage media and program instructions collectively stored on the one or more computer-readable storage media, the program instructions executable by a processor to cause the processor to initiate operations”, the mention being “…a mention of a computer application component…”, the free-form text being “…a textual description of a technology stack of the computer application…”, and the knowledge base being “…a knowledge base of computer components…”
However, analogous art, SINGH, does teach “A computer program product, the computer program product comprising: one or more computer-readable storage media and program instructions collectively stored on the one or more computer-readable storage media, the program instructions executable by a processor to cause the processor to initiate operations”:
([Hardware requirements for machine learning] “The first thing you should determine is what kind of resource does your task requires. Let’s have a look how different tasks will have different hardware requirements:
If your tasks are small and can fit in a complex sequential processing, you don’t need a big system. You could even skip the GPUs altogether. A CPU such as i7–7500U can train an average of ~115 examples/second. So, if you are planning to work on other ML areas or algorithms, a GPU is not necessary.”) This describes the need for a system with some type of processor, a GPU or CPU, in order to train and use machine learning models.
And further:
([Memory & Storage] “Machine learning models (a computer program product made up of program instructions) are getting larger, requiring more memory and storage capacity. High-bandwidth memory (HBM) and solid-state drives (SSDs) are becoming critical for training and running these models efficiently.”) This describes the explicit need for computer-readable storage media to run machine learning models.
It would be obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to combine the base reference of WU with the teachings of SINGH because WU teaches the use of machine learning in is methods, while SINGH teaches the requirements for machine learning methods.
One of ordinary skill in the art would be motivated to do so because without meeting the requirements of any given software, the operations of that software will not function.
WU in view of SINGH still fails to explicitly teach the mention being “…a mention of a computer application component…”, the free-form text being “…a textual description of a technology stack of the computer application…”, and the knowledge base being “…a knowledge base of computer components…”
However, as the specification of the present application admits at [0004-0005], “Application modernization” is a known field for “updating of a legacy application”, that “typically” (This word is used verbatim, showing it is well-known) involves obtaining information from “the legacy application’s technology stack”, which is “usually” (This word is used verbatim, showing it is well-known) in the form of “free-form textual descriptions.” Further, it admits the existence of these “mentions” of computer application components in the “technology stack” of the computer application, and the known aspect of “entity standardization”, explicitly stating “The same application component may be mentioned in different ways in different technology stacks. This poses a significant challenge to entity standardization, which aims to map various technical mentions to entities in a knowledge base. The knowledge base entities are words, phrases, acronyms, symbols, or the like, that are widely recognized by working professionals as referring to specific application components (including computer components). Entities drawn from a knowledge base are thus standard references to application components. A mention extracted from a textual description, however, may take form – name, acronym, symbols, or the like – that differs from the name, acronym, or symbols that are widely recognized by computer professionals. This poses a significant challenge to mapping the mention to the correct entity, which is the aim of standardization.”
Given the specification’s admitted prior art and the disclosure of WU, it would be obvious to one of ordinary skill in the art to combine the methods taught in WU in view of SINGH with the known field of use described in the present specification, prior to the effective filing date of the claimed invention, and that doing so would achieve the claimed invention.
One of ordinary skill in the art would be motivated to do so because, as the present specification points out at [0004], “The automatic extraction of such information from a textual description of a technology stack is a non-trivial task”, and as WU points out in its Abstract, “We also show that bi-encoder linking is very fast with nearest neighbour search (e.g. linking with 5.9 million candidates in 2 milliseconds), and that much of the accuracy gain from the more expensive cross-encoder can be transferred to the bi-encoder via knowledge distillation.”
Further, WU in view of Singh and in view of the applicant’s admitted prior art fails to explicitly teach “wherein the embedding space encoder is trained using a hybrid batch-all and batch-hard online triplet loss mining”. However, analogous art, HERMANS, does teach this:
([2. Learning Metric Embeddings, the Triplet Loss, and the Importance of Mining] “The goal of metric embedding learning is to learn a function… which maps semantically similar points from the data manifold… onto metrically close points…
… A major caveat of the triplet loss, though, is that as the dataset gets larger, the possible number of triplets grows cubically, rendering a long enough training impractical. To make matters worse, f0 relatively quickly learns to correctly map most trivial triplets, rendering a large fraction of all triplets uninformative. Thus mining hard triplets becomes crucial for learning…
In a classical implementation, once a certain set of B triplets has been chosen, their images are stacked into a batch of size 3B… Given the fact that there are up to 6B2 - 4B possible combinations of these 3B images that are valid triplets, using only B of them seems wasteful. With this realization, we propose an organizational modification to the classic way of using the triplet loss: the core idea is to form batches by randomly sampling P classes (person identities), and then randomly sampling K images of each class (person)… Now, for each sample a in the batch, we can select the hardest positive and the hardest negative samples within the batch when forming the triplets for computing the loss, which we call Batch Hard…
…This new formulation of sampling a batch immediately suggests another alternative, that is to simply use all possible PK(PK - K)(K - 1) combinations of triplets, which corresponds to the strategy chosen in [9] and which we call Batch All”) This citation shows that HERMANS simultaneously uses both Batch-Hard and Batch-All online triplet loss mining.
It would be obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to combine the base reference of WU in view of the admitted prior art with the teachings of HERMANS because both references use triplet-loss mining.
One of ordinary skill in the art would be motivated to do so because, as pointed out by HERMANS’ conclusion, “We propose a variant (of triplet loss) that no longer requires offline hard negative mining at almost no additional cost. Combined with a pretrained network, we set the new state-of-the-art on three of the major ReID datasets.”
Claims 16 and 18:
Regarding claims 16 & 18, WU in view of SINGH & the applicant’s admitted prior art and Hermans teaches the limitations of claim 14. Further, claims 16 & 18 comprise similar additional limitations as claims 10 & 12, respectively, and are rejected under the same rationale.
Claim 20:
Regarding claim 20, WU in view of SINGH & the applicant’s admitted prior art and Hermans teaches the limitations of claim 14. Further, WU teaches “mining a plurality of online databases for textual descriptions… compiling the textual descriptions in the knowledge base”:
([5.1 Datasets, TACKBP-2010] “…There are 1,074 and 1,020 annotated mention/entity pairs derived from 1,453 and 2,231 original news and web documents on training and evaluation dataset, respectively.”)
Further, WU teaches “encoding entities extracted from the textual descriptions in the multi-dimensional embedding space”:
([4.1 Bi-encode. Entity Modeling] “The entity representation… is also composed of word-pieces of the entity title and description (for Wikipedia entities, we use the first ten sentences as description). The input to our entity model is: [CLS] title [ENT] description [SEP] where title, description are word-pieces tokens of entity title and description, and [ENT] is a special token to separate entity title and description representation.”) Here we see entities extracted from the textual descriptions encoded into the embedding space. The “Dense Space” is used to embed the encoded representations in at least two dimensions, as shown in the figure, making it a multi-dimensional embedding space.
Further, in light of the applicant’s admitted prior art, it teaches the textual descriptions being “…of computer application components” at [0004-0005], “Application modernization” is a known field for “updating of a legacy application”, that “typically” (This word is used verbatim, showing it is well-known) involves obtaining information from “the legacy application’s technology stack”, which is “usually” (This word is used verbatim, showing it is well-known) in the form of “free-form textual descriptions.” Further, it admits the existence of these “mentions” of computer application components in the “technology stack” of the computer application, and the known aspect of “entity standardization”, explicitly stating “The same application component may be mentioned in different ways in different technology stacks. This poses a significant challenge to entity standardization, which aims to map various technical mentions to entities in a knowledge base. The knowledge base entities are words, phrases, acronyms, symbols, or the like, that are widely recognized by working professionals as referring to specific application components. Entities drawn from a knowledge base are thus standard references to application components. A mention extracted from a textual description, however, may take form – name, acronym, symbols, or the like – that differs from the name, acronym, or symbols that are widely recognized by computer professionals. This poses a significant challenge to mapping the mention to the correct entity, which is the aim of standardization.”
Claims 9, 13, 15, & 19 are rejected under 35 U.S.C. 103 as being unpatentable over WU in view of SINGH & the applicant’s admitted prior art and Hermans, as applied to claims above, and further in view of REIMERS.
Claim 9:
Regarding claim 9, WU in view of SINGH & the applicant’s admitted prior art and Hermans teaches the limitations of claim 8. Further, WU in view of SINGH & the applicant’s admitted prior art fails to explicitly teach “wherein the machine learning model implemented by the embedding space encoder is a Siamese neural network”. However, analogous art, REIMERS, does teach this:
([Abstract, paragraphs 2-3] “…In this publication, we present Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT. We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning tasks, where it outperforms other state-of-the-art sentence embeddings methods.”)
It would be obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to combine the base reference of WU in view of SINGH & the applicant’s admitted prior art with the teachings of REIMERS because both references involve extracting components of free form text and matching them with knowledge in a database.
One of ordinary skill in the art would be motivated to do so because, as pointed out in REIMERS’ abstract above, “This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT” and “…it outperforms other state-of-the-art sentence embeddings methods.”
Claim 13:
Regarding claim 13, WU in view of SINGH & the applicant’s admitted prior art and Hermans teaches the limitations of claim 12. Further, WU in view of SINGH & the applicant’s admitted prior art fails to explicitly teach “wherein the vectorial distance is determined based on a cosine similarity between the encoded representation of the mention and the encoded representation of the entity in the knowledge base”. However, analogous art, REIMERS, does teach this:
([Page 1, Introduction, Col. 2, Last paragraph] “…we developed SBERT. The siamese network architecture enables that fixed-sized vectors for input sentences can be derived. Using a similarity measure like cosine-similarity or Manhatten / Euclidean distance, semantically similar sentences can be found. These similarity measures can be performed extremely efficient on modern hardware, allowing SBERT to be used for semantic similarity search as well as for clustering.”)
It would be obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to combine the base reference of WU in view of SINGH & the applicant’s admitted prior art with the teachings of REIMERS because both references involve extracting components of free form text and matching them with knowledge in a database.
One of ordinary skill in the art would be motivated to do so because, as pointed out in REIMERS’ abstract above, “This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT” and “…it outperforms other state-of-the-art sentence embeddings methods.”
Claims 15 and 19:
Regarding claims 15 & 19, WU in view of SINGH & the applicant’s admitted prior art and Hermans teaches the limitations of claim 14 & 18, respectively. Further, claims 15 & 19 comprise similar additional limitations as claims 9 & 13, respectively, and are rejected under the same rationale.
Claims 21, 23, and 24 are rejected under 35 U.S.C. 103 as being unpatentable over WU, in view of HERMANS, and further in view of Ghosh et al., US Publication 2022/0004366 (“Ghosh”)
Claim 21:
Regarding claim 21, WU teaches “A method, comprising: generating, with a processor, a stand-alone encoder, wherein the stand-alone encoder implements a contrastive learning model”:
([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
As shown in Figure 1, the model uses a stand-alone Cross-encoding Bi-encoder. Contrastive learning, as defined in the art, is a machine learning method that learns by comparing pairs of data to pull similar examples together and push dissimilar ones apart in a representation space, without relying on explicit labels. As shown in the above figure, the extracted mention “Jaguar” is tested to see if it more closely relates to a junior roller coaster or the luxury vehicle brand, by pulling the similar examples together in the “dense space” (a representation space) and pushing the dissimilar ones apart, such as “Ferrari”. It is shown calculating a 0.8 score for the junior roller coaster and a 0.2 score for the luxury vehicle brand, and thus, mapping it as the junior roller coaster, using this method as opposed to labelled data. In doing so, figure 1 illustrates the exact process of contrastive learning.)
Further, WU teaches “encoding, by the stand-alone encoder, a plurality of entities drawn from a knowledge base, wherein the encoding creates a unique encoded representation of each of the plurality of entities”:
([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
As mentioned above, we see the process of contrastive learning result in it mapping the encoded representation of the mention (“Jaguar”) to an entity (“a junior roller coaster”) in the multi-dimensional embedding space (“dense space”).
And further:
([3. Definition and Task Formulation, Entity Linking] “Given an input text document D = {w1, …, wr} and a list of entity mentions MD = {m1, …, mn}, the output of an entity linking model is a list of mention-entity pairs {(mi, ei)}
PNG
media_image2.png
17
45
media_image2.png
Greyscale
where each entity is an entry in a knowledge base (KB) (e.g. Wikipedia),
PNG
media_image3.png
23
46
media_image3.png
Greyscale
. We assume that the title and description of the entities are available, which is a common setting in entity linking (Ganea and Hofmann, 2017; Logeswaran et al., 2019). We also assume each mention has a valid gold entity in the KB, which is usually referred as in-KB evaluation. We leave the out-of-KB prediction (i.e. nil prediction) to future work.”)
WU fails to explicitly teach the model being “trained using a hybrid batch-all and batch-hard online triplet mining”. However, analogous art, HERMANS, does teach this:
([2. Learning Metric Embeddings, the Triplet Loss, and the Importance of Mining] “The goal of metric embedding learning is to learn a function… which maps semantically similar points from the data manifold… onto metrically close points…
… A major caveat of the triplet loss, though, is that as the dataset gets larger, the possible number of triplets grows cubically, rendering a long enough training impractical. To make matters worse, f0 relatively quickly learns to correctly map most trivial triplets, rendering a large fraction of all triplets uninformative. Thus mining hard triplets becomes crucial for learning…
In a classical implementation, once a certain set of B triplets has been chosen, their images are stacked into a batch of size 3B… Given the fact that there are up to 6B2 - 4B possible combinations of these 3B images that are valid triplets, using only B of them seems wasteful. With this realization, we propose an organizational modification to the classic way of using the triplet loss: the core idea is to form batches by randomly sampling P classes (person identities), and then randomly sampling K images of each class (person)… Now, for each sample a in the batch, we can select the hardest positive and the hardest negative samples within the batch when forming the triplets for computing the loss, which we call Batch Hard…
…This new formulation of sampling a batch immediately suggests another alternative, that is to simply use all possible PK(PK - K)(K - 1) combinations of triplets, which corresponds to the strategy chosen in [9] and which we call Batch All”) This citation shows that HERMANS simultaneously uses both Batch-Hard and Batch-All online triplet loss mining.
It would be obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to combine the base reference of WU with the teachings of HERMANS because both references use triplet-loss mining.
One of ordinary skill in the art would be motivated to do so because, as pointed out by HERMANS’ conclusion, “We propose a variant (of triplet loss) that no longer requires offline hard negative mining at almost no additional cost. Combined with a pretrained network, we set the new state-of-the-art on three of the major ReID datasets.”
Ghosh teaches or suggests wherein the one or more plurality of entities comprises a textual identifier of an application component (see Fig. 6-12; para. 0070 - source code for each of the plurality of available features. The saved source code provides a repository of source code for available features that the platform can use to generate object code for the actual application (e.g., based on the selected operating system) and/or to send the source code to select third party software developers to perform desired integration or modification; para. 0084 - Features are the atomic unit of a real-world application that provide functionality to the app; para. 0086 - templates are very well structured collection of features that form a real-world application; para. 0099 - Nodes are not the source or object code for that particular features but represent that entity in name and values of the attributes that are associated with it. If desired, each node can be linked to its corresponding source code (file) or object code (file); para. 0101 – wide range of nodes and edges comprising a wide range of software development entities and relationships on the software development platform; para. 0104 - knowledge graph is likely in practical implementation in constant evolution, which means that relevant embeddings are themselves in constant evolution; para. 0107 - platform can determine if the knowledge graph changes also affect the embeddings required for the application or service (e.g. new entities added; para. 0110 – package software components and method(s) that implement this capability should be very simple to use and preferably only requires as input the algorithm name and a hyperparameter; para. 0112 - embedding can also be loaded into memory or tables that are not graph databases which are easier to store and perform mathematical operations on; para. 0117 - software development platform can be configured to store a plurality of different hyper parameters corresponding to different software tools (services, software clients such as feature recommendation) on the platform. As explained above, the software development platform can be configured to control the embedding generation with the hyper parameters and generate different embeddings and store the different embeddings corresponding to different software tools; para. 0118 - query ( a query seeking information from graph database) is applied to the embedding to generate an output for the control of an operation, display, or recommendation; para. 0119 - architecture can also be implemented to control the arrangement and/or connections between features in a software project or other design or development functions using the graph database and/or embedding technology; para. 0123 - A feature of the software application is a function or set of functions for the application; para. 0128 - templates 550 are grouped by types.).
Accordingly, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Ghosh with those of Wu. One would have been motivated to do so for the purpose of efficiently maintaining a database of entities with corresponding identifiers and generating corresponding embeddings for use in efficiently responding to inputs, improving entity maintenance and interaction, as taught by Ghosh (0112, 0117, and 0118).
Claim 23:
Regarding claim 23, WU in view of HERMANS in view of Ghosh teaches the limitations of claim 21. Further, WU teaches “wherein a backbone of the machine learning model is a context-sensitive, per-trained transformer language model”:
([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
As shown in the figure, a Bi-Encoder and Cross-Encoder are both used, which are both examples of deep pre-trained bidirectional transformers used in context-sensitive language modeling, as is also supported in Section 2, Related Work “…Humeau et al. (2019) studied different architectures to use deep pre-trained bidirectional transformers and performed detailed comparison of three different architectures, namely bi-encoder, poly-encoder, cross-encoder on tasks of sentence selection in dialogues. Inspired by their work, we use similar architectures to the problem of entity linking, and in addition, demonstrate that bi-encoder can be a strong model for retrieval. Instead of using the poly-encoder as a trade-off between cross-encoder and bi-encoder, we propose to train a bi-encoder model with knowledge distillation (Buciluundefined et al., 2006; Hinton et al., 2015) from a cross-encoder model to further improve the bi-encoder’s performances.”
Claim 24:
Regarding claim 24, WU in view of HERMANS in view of Ghosh teaches the limitations of claim 21. Further, WU teaches “encoding, with the stand-alone encoder, an entity added to existing entities of the knowledge base; wherein no further training of the stand-alone encoder is necessary in response to the encoding of the entity added to the existing entities of the knowledge base”:
([Figure 1]
PNG
media_image1.png
430
836
media_image1.png
Greyscale
As mentioned above, we see the process of contrastive learning result in it mapping the encoded representation of the mention (“Jaguar”) to an entity (“a junior roller coaster”) in the multi-dimensional embedding space (“dense space”).
And further:
([3. Definition and Task Formulation, Entity Linking] “Given an input text document D = {w1, …, wr} and a list of entity mentions MD = {m1, …, mn}, the output of an entity linking model is a list of mention-entity pairs {(mi, ei)}
PNG
media_image2.png
17
45
media_image2.png
Greyscale
where each entity is an entry in a knowledge base (KB) (e.g. Wikipedia),
PNG
media_image3.png
23
46
media_image3.png
Greyscale
. We assume that the title and description of the entities are available, which is a common setting in entity linking (Ganea and Hofmann, 2017; Logeswaran et al., 2019). We also assume each mention has a valid gold entity in the KB, which is usually referred as in-KB evaluation. We leave the out-of-KB prediction (i.e. nil prediction) to future work.”) The claimed method inherently results in no further training of the encoder being necessary in response to the entity being added to the knowledge database, as once it is added, it can be used in the method which interprets mappings without further training, just as is shown in the citations of WU as well.
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over WU in view of HERMANS, in view of Ghosh, as applied to claims above, and further in view of REIMERS.
Claim 22:
Regarding claim 22, WU in view of HERMANS in view of Ghosh teaches the limitations of claim 21. Further, WU in view of HERMANS fails to explicitly teach “wherein a machine learning model implemented by the stand-alone encoder is a Siamese neural network”. However, analogous art, REIMERS, does teach this:
([Abstract, paragraphs 2-3] “…In this publication, we present Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT. We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning tasks, where it outperforms other state-of-the-art sentence embeddings methods.”)
It would be obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to combine the base reference of WU in view of HERMANS in view of Ghosh with the teachings of REIMERS because both references involve extracting components of free form text and matching them with knowledge in a database.
One of ordinary skill in the art would be motivated to do so because, as pointed out in REIMERS’ abstract above, “This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT” and “…it outperforms other state-of-the-art sentence embeddings methods.”
Response to Arguments
Rejections under 35 USC 103:
Applicant argues that Hermans is not combinable with Wu because the batch images described in Hermans introduces a change in the basic principle under which the Wu reference was designed to operate, wherein each entity is defined only by a short textual description.
The Examiner respectfully disagrees.
The Examiner notes though Herman teaches the use of image data, Herman also describes the conversion of the image data into embeddings where they can be compared to other embeddings and taking into account features and corresponding semantics (see at least Hermans 1. Introduction - triplet loss optimizes the embedding space such that data points with the same identity are closer to each other than those with different identities. A visualization of such an embedding is shown in Figure 1. Fig. 1 - triplet loss learns semantically meaningful features.). The Examiner notes that using embeddings and taking into account features and corresponding semantics sufficiently corresponds to the textual data and how it is used in Wu. Accordingly, the concepts described Hermans is combinable with Wu.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Andrew T McIntosh whose telephone number is (571)270-7790. The examiner can normally be reached M-Th 8:00am-5:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tamara Kyle can be reached at 571-272-4241. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW T MCINTOSH/Primary Examiner, Art Unit 2144
Prosecution Insights