DETAILED ACTIONS
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim this application being in benefit of foreign priority from Taiwan Patent Application No. TW113117935.5 filed on May 15th, 2024.
Information Disclosure Statement
The information disclosure statement (“IDS”) filed on 04/17/2025 was reviewed and the listed references were noted.
Drawings
The 8-page drawings have been considered and placed on record in the file.
Status of Claims
Claims 1-11 are pending.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-11 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1,
Step 1 Analysis: Claim 1 is directed to a system, which falls within one of the four statutory categories. (Step 1: YES)
Step 2A-Prong 1 Analysis: The limitation of “to recognized image blocks and text blocks on the specification, wherein the image blocks have corresponding covering ranges”, as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind which falls within the Mental Process grouping of abstract ideas. For example, a human can recognize the image blocks and text blocks on a printed Specification or technical drawing. Similarly, the limitation of “to assign preference values to the text blocks for matching with the image blocks based on position relationships between the text blocks and the image blocks, and content characteristics of the text blocks”, as drafted, is a process that under its broadest reasonable interpretation, covers performance of the limitation in the mind which also falls within the Mental Process grouping of abstract ideas. As an example, a human can read the Specification or technical drawing and determine the matching text blocks with the image blocks based on their position on the printed technical drawing. Accordingly, the claim recites an abstract idea.
Step 2A-Prong 2 Analysis: The limitation of receiving a specification is considered to be an insignificant extra-solution activity for mere data gathering. An initial step of receiving a specification does not integrate the exception into a practical application or add significantly more. The claim does not include additional elements that amount to an integration of the judicial exception into a practical application, nor to significantly more than the judicial exception. The claim is not patent eligible. (Step 2A: YES)
Step 2B Analysis: Because the claim fails under Step 2A, the claim is further evaluated under Step 2B. The claim herein does not contain additional elements that are sufficient to amount to significantly more than the judicial exception, because as discussed above with respect to integration of the abstract idea into a practical application, the additional element/limitation “receiving a specification” amounts to no more than an insignificant well-understood, routine, and conventional element. Therefore, independent claim 1 is not patent eligible. (Step 2A: YES)
Regarding dependent claims 2-11, they do not overcome the deficiencies of the rejected independent claim 1, and they are also rejected.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-4 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Gardner et al., (US 2024/0119751 A1, published 04/11/2024), hereinafter referred to as Gardner.
Claim 1
Gardner discloses a matching system for images and text descriptions in a specification (Gardner, Fig. 2), comprising:
an image-and-text recognition device (Gardner, [0030], “The software may be divided into local software 210 that executes on one or more computing devices local to an end-user (collectively “local devices”) and, in some cases, cloud-based software 212 that is executed on one or more computing devices remote from the end-user (collectively “cloud computing devices”) accessible via a network (e.g., the Internet)”), configured to receive a specification (Gardner, [0031], “At step 310, the schematic diagram data extraction application 200 accesses the schematic diagram (e.g., P&ID) in an image-only format (e.g., a JPG, PNG, BMP, TIFF, PDF, etc.)”), and to recognize image blocks (Gardner, [0031], “set of texts labels, set of symbols, and set of links that have been extracted from the schematic diagram”, [0005], machine learning (ML) techniques and that utilize trained ML models to identify symbols and links and determine their properties, Fig. 4A, [0039], “the bounding box 420 of the symbol”) and text blocks (Gardner, [0031], “set of texts labels, set of symbols, and set of links that have been extracted from the schematic diagram” [0005], optical character recognition (OCR) techniques to identify text labels and recognize the text characters, Fig. 4A, [0039], “bounding box 410 of the text label”) on the specification (Gardner, Fig. 1A);
wherein the image blocks (Gardner, [0039], the bounding box 420 of the symbol) have corresponding covering ranges (Gardner, [0037], “a “context” refers a region of the schematic diagram that encloses a possible text label and symbol pair, as well as surroundings 430 about both the text label and symbol. A context may take the form of an image of the region whose size is defined by constructing a bounding box around the bounding box of the text label and the bounding box of the symbol, and adding an amount of padding (e.g., 80 pixels) to capture surroundings”, [0039], “The region defined by the context encloses the bounding box 410 of the text label and the bounding box 420 of the symbol, as well as surroundings 430 about both these bounding boxes); and
a preference-value calculation device (Gardner, Fig. 2), configured to assign preference values (Gardner, Fig. 3, step 330, For each possible text label and symbol pair, apply context and request to symbol association ML Model, [0047], “once all possible text label and symbol pair have been processed and association decisions rendered, the schematic diagram data extraction application 200 selects associations between text labels and symbols therefrom based on the scores, to produce text label to symbol associations. In one implementation, the selection may include a maximum selection where, for a given text label, the text label and symbol pair with the score indicating the greatest confidence is selected as the association. The selection may also include a configurable minimum confidence threshold (e.g., 0.5), which excludes text label and symbol pairs when the score indicates a low confidence of association.”, the score is analogous to the preference values) to the text blocks for matching with the image blocks based on position relationships between the text blocks and the image blocks (Gardner, [0012], “Both models are trained to learn a plurality of graphical cues (e.g., including, but not limited to, distance) that indicate a text label references a particular symbol or link”, [0013], “The schematic diagram data extraction application, for each possible text label and symbol pair, applies to a symbol association ML model a context that describes a region of the schematic diagram surrounding the possible text label and symbol pair and a request that precisely designates the possible text label and symbol pair, to produce a score indicating confidence in association. The schematic diagram data extraction application selects associations between text labels and symbols based on the scores.”, [0038], “a “context” refers a region of the schematic diagram that encloses a possible text label and symbol pair, as well as surroundings 430 about both the text label and symbol. A context may take the form of an image of the region whose size is defined by constructing a bounding box around the bounding box of the text label and the bounding box of the symbol, and adding an amount of padding (e.g., 80 pixels) to capture surroundings. If the context is less than a minimum size (e.g., 256×256 pixels) it may be expanded to meet the minimum size. Further, if the context extends beyond the borders of the schematic diagram, its center may be adjusted so it falls fully within the schematic diagram.”), and content characteristics of the text blocks (Gardner, Fig. 3, step 330, for each possible text label and symbol pair, apply context and request to symbol association ML model, [0039], “The surroundings provide graphical cues (e.g., presence of an arrow, presence of other nearby symbols, links, or text labels, etc.) that would not be available if only a crop to the bounding box 410 of the text label and the bounding box 420 of the symbol were provided.”).
Claim 2
Gardner discloses the matching system as claimed in claim 1 (Gardner, Fig. 2, Fig. 3), wherein the preference-value calculation device (Gardner, Fig. 2) is further configured to:
calculate overlapping areas of the text blocks and the covering range of a first image block among the image blocks (Gardner, Fig. 4A), and obtain area scores of the text blocks which overlap the first image block (Gardner, [0037], “a “context” refers a region of the schematic diagram that encloses a possible text label and symbol pair, as well as surroundings 430 about both the text label and symbol. A context may take the form of an image of the region whose size is defined by constructing a bounding box around the bounding box of the text label and the bounding box of the symbol, and adding an amount of padding (e.g., 80 pixels) to capture surroundings”, [0044], “ The association decision may take the form of a score indicating confidence in association between the text label and symbol pair, which may be mathematically represented as a.sub.k for pair C.sub.k with a value of 1 indicating they are definitely associated and a value of 0 indicating they are definitely not associated.”);
for the text blocks which overlap the first image block, calculate distances from the first image blocks to obtain distance scores of the text blocks (Gardner, [0034], “the heuristics process 236 may exclude pairs from consideration where the text label is under a threshold distance of a symbol. When the bounding box of a given text label is under a threshold distance of the bounding box of a given symbol, and the given text label and the given symbol mutually see each other as the closest region (i.e. among all the symbols, the closest to the given text label is the given symbol, and among all text labels, the closest to the symbol is the given text label), the heuristics process 236 may automatically associate the given text label and the given symbol, and exclude the given text label and given symbol pair from ML model processing.”);
based on positions and descriptive forms of the text blocks which overlap the first image block, and a frame range of the first image block, determine negative scores of the text blocks which overlap the first image block (Gardner, [0045], “the symbol association ML model 232 may use a loss function that includes a penalty to associations that is quadratically increasing with distance. There is often an exponential distribution in schematic diagrams of text label to associated symbol distances, such that there are many associations where text labels and symbols are close, and only a few where they are not. A penalty to associations that is quadratically increasing with distance may force the model to consider the far associations despite their relatively low frequency of occurrence, while keeping the loss function smooth (i.e., without abrupt “steps” at certain distances).”); and
based on the area scores, distance scores and negative scores, calculate the preference values of the text blocks which overlap the first image block (Gardner, [0036], “for each possible text label and symbol pair in C, the schematic diagram data extraction application 200 constructs a context and a request, and applies the context and request to the symbol association ML 232 model to produce a score indicating confidence in association.”).
Claim 3
Gardner discloses the matching system as claimed in claim 2 (Gardner, Fig. 2, Fig. 3), wherein the preference-value calculation device is further configured to add the text blocks having preference values that satisfy a selection condition to a selection list of the first image block (Gardner, [0047], “At step 340, once all possible text label and symbol pair have been processed and association decisions rendered, the schematic diagram data extraction application 200 selects associations between text labels and symbols therefrom based on the scores, to produce text label to symbol associations. In one implementation, the selection may include a maximum selection where, for a given text label, the text label and symbol pair with the score indicating the greatest confidence is selected as the association. The selection may also include a configurable minimum confidence threshold (e.g., 0.5), which excludes text label and symbol pairs when the score indicates a low confidence of association.”).
Claim 4
Gardner discloses the matching system as claimed in claim 3 (Gardner, Fig. 2, Fig. 3), further comprising a filtering device (Gardner, Fig. 2); wherein when a first text block in the selection list of the first image block is associated with a second image block of the image blocks, the filtering device determines whether to keep or delete the first text block in the selection list of the first image block based on the distance scores, the area scores, and coverage points (Gardner, [0032, “. As part of such determination, the heuristics process 236 may perform pre-processing to exclude certain pairs from consideration by the symbol association ML model 232, such that the determined possible text label and symbol pairs are less than all pairs. Such exclusion may improve performance by reducing the number of pairs requiring ML model processing, with “easy” cases instead automatically assumed to be associated or not associated.”).
Allowable Subject Matter
Claims 5 and 7 are objected to as being dependent upon a rejected base claim, but would be allowable if the 101 rejections are overcome and if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: The claimed features such as “wherein the coverage point is expressed by the formula:
PNG
media_image1.png
132
589
media_image1.png
Greyscale
wherein p represents the number of image blocks, k represents a total number of all the text blocks which overlap the image blocks p, and 𝐴𝑟𝑒𝑎_𝑠𝑐𝑜𝑟𝑒𝑝𝑡 corresponds to a total overlapping area of k text blocks which overlap the image blocks p” claimed in dependent claim 5, in combination with the remainder of the limitations of the claims, are neither anticipated nor obvious in view of the prior art of record.
In the closest prior art of record, Gardner (US 2024/0119751 A1), teaches a covering range in a form of a bounding box around both the symbol bounding box which is analogous to the image block and the text bounding box as shown in Fig. 4A which is used to calculate a score to determine the pairing of the symbol and the text.
However, Gardner fails to teach that the coverage point us expressed by the formula wherein p represents the number of image blocks, k represents a total number of all the text blocks which overlap the image blocks p, and 𝐴𝑟𝑒𝑎_𝑠𝑐𝑜𝑟𝑒𝑝𝑡 corresponds to a total overlapping area of k text blocks which overlap the image blocks p. Therefore claim 5 would be allowable for claiming the limitation “wherein the coverage point is expressed by the formula:
PNG
media_image1.png
132
589
media_image1.png
Greyscale
wherein p represents the number of image blocks, k represents a total number of all the text blocks which overlap the image blocks p, and 𝐴𝑟𝑒𝑎_𝑠𝑐𝑜𝑟𝑒𝑝𝑡 corresponds to a total overlapping area of k text blocks which overlap the image blocks p” in combination with the remainder of the limitations of the claims.
Because the cited prior art of records does not teach or suggest each and every feature of dependent claim 1, this claim would be allowable. Claim 7 would be allowable by virtue of their dependency on dependent claim 5.
Claims 6 and 8-9 are objected to as being dependent upon a rejected base claim, but would be allowable if the 101 rejections are overcome and if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: The claimed features such as “when the distance scores of the first text block corresponding to the first image block and the second image block respectively are the same, the filtering device further compares the area scores of the first text block corresponding to the first image block and the second image block respectively, and when the area score of the first text block corresponding to the first image block is lower, the first text block is deleted from the selection list of the first image block and when the area scores of the first text block corresponding to the first image block and the second image block respectively are the same, the filtering device further compares the coverage points of the first text block corresponding to the first image block and the second image block respectively, in addition, when the coverage point of the first text block is lower, the first text block is kept in the selection list of the first image block, and when the coverage point of the first text block is higher, the first text block is deleted from the selection list of the first image block” claimed in dependent claim 6, in combination with the remainder of the limitations of the claims, are neither anticipated nor obvious in view of the prior art of record.
In the closest prior art of record, Gardner (US 2024/0119751 A1), teaches a heuristic process to determine which pairing of text and symbols can be removed from potential pairings, ([0048], “the heuristics process 236 may exclude any pairs from consideration where the link is greater than a maximum distance from the text label (e.g., outside a radius extending from the center of the bounding box of the text label). Such exclusions may remove pairs from consideration by the link association ML model 234 that are unlikely from being associated, saving processing resources and avoiding potential false positives.”).
However, Gardner fails to teach filtering the pairing by calculating the distance scores between the text block and two image blocks and determining that they are the same. Also, Gardner fails to teach comparing the area scores of the first text block corresponding to the first symbol and the second symbol, and when the area score of the first text block corresponding to the first symbol is lower, the first text block is deleted from the selection list of the first image block and when the area scores of the first text block corresponding to the first image block and the second symbol respectively are the same, it compares the coverage points of the first text block corresponding to the first symbol and the second symbol respectively, in addition, when the coverage point of the first text block is lower, the first text block is kept in the selection list of the first symbol, and when the coverage point of the first text block is higher, the first text block is deleted from the selection list of the first symbol. Therefore claim 6 would be allowable for claiming the limitation “when the distance scores of the first text block corresponding to the first image block and the second image block respectively are the same, the filtering device further compares the area scores of the first text block corresponding to the first image block and the second image block respectively, and when the area score of the first text block corresponding to the first image block is lower, the first text block is deleted from the selection list of the first image block and when the area scores of the first text block corresponding to the first image block and the second image block respectively are the same, the filtering device further compares the coverage points of the first text block corresponding to the first image block and the second image block respectively, in addition, when the coverage point of the first text block is lower, the first text block is kept in the selection list of the first image block, and when the coverage point of the first text block is higher, the first text block is deleted from the selection list of the first image block”, in combination with the remainder of the limitations of the claims.
Because the cited prior art of records does not teach or suggest each and every feature of dependent claim 6, this claim would be allowable. Claims 8-9 would be allowable by virtue of their dependency on dependent claim 6.
Claims 10-11 are objected to as being dependent upon a rejected base claim, but would be allowable if the 101 rejections are overcome and if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: The claimed features such as “the covering range corresponding to the image block is arranged with the image block as the center, horizontally extending a first predetermined number of basic units to the left and right respectively, and on the upper and lower sides of the image block horizontally extending a second predetermined number of basic units” claimed in dependent claim 10, in combination with the remainder of the limitations of the claims, are neither anticipated nor obvious in view of the prior art of record.
In the closest prior art of record, Gardner (US 2024/0119751 A1), teaches a covering range in a form of a bounding box around both the symbol bounding box which is analogous to the image block and the text bounding box as shown in Fig. 4A. The bounding box has the image block or the symbol bounding box inside it.
However, Gardner fails to teach that the covering range has the symbol as the center and it extends a predetermined number to the left and to the right and a second predetermine number to the upper and lower sides of the symbol bounding box. Therefore claim 10 would be allowable for claiming the limitation “the covering range corresponding to the image block is arranged with the image block as the center, horizontally extending a first predetermined number of basic units to the left and right respectively, and on the upper and lower sides of the image block horizontally extending a second predetermined number of basic units”, in combination with the remainder of the limitations of the claims.
Because the cited prior art of records does not teach or suggest each and every feature of dependent claim 10, this claim would be allowable. Claim 11 would be allowable by virtue of their dependency on dependent claim 10.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Gardner et al., (US 20220044146 A1) – teaches recognizing text blocks and symbol blocks and determining overlap between them.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DENISE G ALFONSO whose telephone number is (571)272-1360. The examiner can normally be reached Monday - Friday 7:30 - 5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached at (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DENISE G ALFONSO/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662
Prosecution Insights