CTFR 18/502,688 CTFR 84417 DETAILED ACTION Case Status This office action is in response to remarks and amendments of 11 March 2026. Claims 1, 3-9 and 21-23 have been examined. Claims 11-15, 17-20 are withdrawn. Pertinent Prior Art 07-96 AIA The prior art made of record and not relied upon is considered pertinent to applicant's disclosure : 20140122549 Abstract Intermediate steps display for a calculator 5584699 Fig.’s 5-8 Teaching geometry proofs and steps thereof 20140293030 Pars. 21-22 Solve a math homework problem with mobile device camera 20120141032 Pars. 37, 56, 105, 117 Searching using objects recognized from hand-drawn diagrams including electrical circuits, chemical compositions, biological compositions, mechanical, etc. 20220164548 Pars. 39-44 Multi-modal image and text search queries including observing image machine learning based embeddings as vectorized dimensional representations 20220130499 Pars. 36-41 Machine learning based visual question answering including searching for results based on an input comprising a question and an image 20080104153 Pars. 97-99 Step-by-step mathematical expression solver 20230418861 Fig. 11B Diagram/sketch with text-based searching including determining a representation of placement, sizes, and shapes of sketched diagram objects 20210233140 Fig. 3 Converting image features in to formal descriptors for image-based searches 20210073530 Abstract, fig. 4 Machine learning classification of objects, including text and symbols, within a handwritten diagram 20200279008 Pars. 32-34 Product search results based on gesture input received via a digital canvas 20210158419 Pars. 22-24 User interaction with visual diagrams for search DATE, N et al. An efficient approach to automated geometry diagram parsing. JEI, Vol 5, Oct 2022, pp. 1-8 [online], [retrieved 2025-06-20]. Retrieved from the Internet <URL:https://emerginginvestigators.org/articles/21-255> Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-23-aia AIA The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows: 1. Determining the scope and contents of the prior art. 2. Ascertaining the differences between the prior art and the claims at issue. 3. Resolving the level of ordinary skill in the pertinent art. 4. Considering objective evidence present in the application indicating obviousness or nonobviousness. 07-20-02-aia AIA This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention. 07-21-aia AIA Claim s 1, 3, 5-9, and 21-23 are rejected under 35 U.S.C. 103 as being unpatentable over Costabello et al., Pub. No.: US 20200356829 A1, hereinafter Costabello in view of Lu et al., Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning , hereinafter Lu . As per claim 1, Costabello discloses A computer-implemented method for returning a search result, the method comprising: receiving a search request from a user, the search request including an image that depicts a diagram with at least one associated question (fig. 6, item 610; also, fig. 1 shows a simple line-based graphic representation of objects conveying at least one or more of particular positional, pose, size, and spatial relationships/arrangements thereof (i.e., image that depicts a diagram) along with an associated question); processing the search request by analyzing the at least one associated question and the diagram included in the search request using a machine-learned diagram parsing model to perform geometric entity recognition of the diagram and generate a portion of a formal language representation of the diagram (pars. 22, 27-28, 33-35, 66 disclose model-based recognition of more than mere lines and points (which are “geometric entities” as exemplified in par. 77 of the published specification) including, but not limited to objects, actions, scenes, other features of the input image based on pixels, edges, lines, blobs, colors, shapes etc.; see at least the abstract and at least pars. 22-26 for multiple alternative disclosures of formal language representation including classifications, coordinate/pixel data, graph, embeddings/vectors, etc.); in response to the performing of the geometric entity recognition of the diagram in the search request and the generating of the portion of the formal language representation of the diagram, obtaining the portion of the formal language representation of the diagram (see rejection above including at least the abstract of Costabello and pars. 22-26) , wherein the formal language representation of the diagram comprises at least one […] rule that is identified based on […] the diagram (note that the diagram requires a particular arrangement that is conveyed as requirements such as a labeled edge in par. 23 and spatial, sequential, possessive and action relationships as disclosed in par. 36, each of which is “included in” one or more of the elements of Costabello mapped to “formal language representation” above, and thus, all of which correspond to one or more “rule” as claimed) . Costabello does not expressly disclose that the rule is a mathematical rule that is identified based on geometric markings within the diagram. However, Lu, in the related field of endeavor of answering questions based on diagram content discloses t his. See at least sections 5.2, 6.1 and Table 13 of Lu which disclose automated detection of varied diagram symbols including relational symbols for parallel, perpendicular, isosceles, and that “ the diagram parser first applies Hough Transformation (Shapiro and Stockman, 2001) to extract geometry primitives (points, lines, arcs, and circles) ; fig. 4 shows a relation set defined in formal language including mathematical rules identified based on geometric markings within an inputted diagram such as Perpendicular(Line(B,D),Line(A,D)), PointLiesOnLine(D,Line(A,C)) and Equals(Line(B,C),Line(A,B)) . These align with the examples provided in the last sentence of par. 24 of Applicant’s specification: “For example, based on markings in the diagram, a rule can be generated that one or more line segments have equal length, one or more points lie on the same line segment, one or more line segments are parallel or perpendicular to one another, and the like. ” Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the cited references because Lu would have allowed Costabello’s search system to identify mathematical rules from geometric markings within diagrams and apply those rules in a step-by-step manner to solve for unknown values, thereby improving the relevance and quality of mathematical search results returned to users. See Lu , abstract and section 5.2. providing the formal language representation including the at least one mathematical rule that is identified based on the diagram to a search engine as a search query ( Costabello pars. 28-40, 38, 41, 72 disclose providing a formal language representation of a diagram to a search engine as a search query; Lu as cited above modifies the content of that formal language representation to include mathematical rules identified from geometric markings in the diagram, such as Perpendicular(Line(B,D),Line(A,D)) derived from perpendicular markings; the combination thus results in a formal language representation containing at least one mathematical rule identified from geometric markings being provided to Costabello’s search engine as a search query, as claimed); and receiving, as a search result, an answer to the at least one associated question ( Costabello, pars. 28-40, 38, 41, 74 disclose generating and receiving an answer search result) the answer comprising a step-by-step guide for reaching at least one mathematical solution of the at least one associated question, wherein the step-by-step guide is generated by applying the at least one mathematical rule identified from the geometric markings to solve for one or more unknown values in the diagram ( Lu , abstract discloses (with emphasis) “Inter-GPS incorporates theorem knowledge as conditional rules and performs symbolic reasoning step by step . Also, a theorem predictor is designed to infer the theorem application sequence fed to the symbolic solver for the more efficient and reasonable searching path” and as per Section 6.2, the output of the SEARCH function of Algorithm 1 is explicitly stated as “Output Numeric goal value g ∗ and theorem application S ” where the algorithm repeatedly appends each applied theorem to S (i.e., step-by-step guide) while solving the problem in at least lines 6, 15 and 23 of the SEARCH function . Claim interpretation note: this limitation does not require a fully written out, natural language step by step solution for an end user. Under BRI, “step-by-step guide” reasonably covers a sequence of applied rules/theorems that guides how the solution is reached even if not written in a natural language textbook style manner). As per claim 3, Costabello as modified discloses the computer-implemented method of claim 1, wherein the geometric entity recognition is performed using at least one of a Hough transformation and a machine- learned object detector (see rejection of claim 1 including at least Costabello pars. 22, 44, 50, 66; see also, Lu , section 5.2). As per claim 5, Costabello as modified discloses The computer-implemented method of claim 1, wherein performing the geometric entity recognition includes identifying one or more geometric features of the diagram (see rejection of claim 1 including at least Costabello pars. 22, 44, 50, 66; see also, Lu as cited in the rejection of claim 1). As per claim 6, Costabello as modified discloses The computer-implemented method of claim 1, wherein the machine-learned diagram parsing model generates at least the portion of the formal language representation of the diagram by performing symbolic detection using a machine-learned symbolic detection model ( Costabello abstract, pars. 26, 34, 42-50, 65-69 all disclose performing symbolic detection as claimed; see also, Lu section 7.1). As per claim 7, Costabello as modified discloses The computer-implemented method of claim 6, wherein the symbolic detection model identifies one or more known symbols in the diagram and outputs at least the portion of the formal language representation of the diagram based on the one or more known symbols ( Costabello pars. 26, 42-45, 49-50, 65-66 all disclose multiple examples of outputting a symbol-based representation as claimed; see also, Lu section 7.1). As per claim 8, Costabello as modified discloses the computer-implemented method of claim 1 wherein the answer comprises the step-by-step guide for reaching the at least one mathematical solution of at least one mathematical problem of the at least one associated question (see rejection of last limitation of claim 1). As per claim 9, Costabello as modified discloses The computer-implemented method of claim 1, wherein the formal language representation of the diagram includes at least one feature of the diagram (see at least Costabello pars. 22-26, 44, 66 for multiple features of the diagram; see also, Lu as cited in the rejection of claim 1). As per claim 21, Costabello as modified discloses The computer-implemented method of claim 1, wherein the at least one mathematical rule comprises a geometric rule utilized to solve a geometric problem in the diagram, and the mathematical solution comprises a geometric solution to the geometric problem based on the geometric rule (see Lu at least abstract, sections 5.2, 6.1 and Table 13). As per claim 22, Costabello as modified discloses The computer-implemented method of claim 1, wherein the at least one mathematical rule comprises a mathematical rule utilized to solve a mathematical problem in the diagram, and the at least one solution comprises a mathematical solution to the mathematical problem based on the mathematical rule (see Lu at least abstract, sections 5.2, 6.1 and Table 13). As per claim 23, Costabello as modified discloses The computer-implemented method of claim 1, wherein the portion of the formal language representation of the diagram being generated by the machine-learned diagram parsing model and then obtained is utilized to, in response to i) the generating of the portion of the formal language representation of the diagram by the machine-learned diagram parsing model, and ii) the obtaining of the portion of the formal language representation of the diagram, identify the answer comprising the step-by-step guide for reaching the at least one mathematical solution of at least one mathematical problem of the at least one associated question (see rejection of claim 1 and rationales to combine references) . 07-21-aia AIA Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Costabello as modified and further in view of Boyer et al., Patent No.: US 5157736 A, hereinafter Boyer . As per claim 4, Costabello as modified discloses The computer-implemented method of claim 1. The combination does not explicitly disclose, however Boyer in the related field of endeavor of image-based information retrieval discloses further comprising performing pre-processing on the diagram to remove one or more artifacts of the diagram before performing the geometric entity recognition ( Boyer , col. 1, lines 50-65 and col. 7, lines 45-60). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the cited references because Boyer would have allowed Costabello as modified to remove diagram artifacts before arriving at a final vectorized representation of the diagram, including correcting defects in the vector representation of the diagram, thereby resulting in a more accurate and correct understanding of a diagram. Response to Arguments Applicant's arguments filed 11 March 2026 have been fully considered; non-patent literature Lu et al., Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning has been applied to address claim amendments. Conclusion 07-40 AIA Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL . See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action. Any inquiry concerning this communication or earlier communications from the examiner should be directed to SYED HASAN whose telephone number is (571)270-5008. The examiner can normally be reached M-F 8am - 5 pm. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Boris Gorney can be reached at (571)270-5626. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /SYED H HASAN/ Primary Examiner, Art Unit 2154 Application/Control Number: 18/502,688 Page 2 Art Unit: 2154 Application/Control Number: 18/502,688 Page 3 Art Unit: 2154 Application/Control Number: 18/502,688 Page 4 Art Unit: 2154 Application/Control Number: 18/502,688 Page 5 Art Unit: 2154 Application/Control Number: 18/502,688 Page 6 Art Unit: 2154 Application/Control Number: 18/502,688 Page 7 Art Unit: 2154 Application/Control Number: 18/502,688 Page 8 Art Unit: 2154 Application/Control Number: 18/502,688 Page 9 Art Unit: 2154 Application/Control Number: 18/502,688 Page 10 Art Unit: 2154 Application/Control Number: 18/502,688 Page 11 Art Unit: 2154 Application/Control Number: 18/502,688 Page 12 Art Unit: 2154 Application/Control Number: 18/502,688 Page 13 Art Unit: 2154 Application/Control Number: 18/502,688 Page 14 Art Unit: 2154
Prosecution Insights