Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This final Office action is in response to applicant’s communication received on June 30, 2026, wherein claims 1-11, 13, 15-19, and 21-23 are currently pending.
Response to Arguments
Applicant's arguments filed have been fully considered but they are directed to the amended claims and newly added limitations to the amended claims. The newly amended claims and the newly added limitations are considered for the first time in the rejection below.
35 USC §101 discussion:
The amendment to the claims do not help the claims overcome the §101 rejection. The amended claims are directed to creating workflow using information received/gathered from people (humans) where the information itself is abstract information created by humans/people (human drawings). The claims actually state that taking human created information and then presenting the information with maybe some modifications (where those modification are abstract data (drawings/etc.,) corrections/manipulations) – diagram conversion where the diagram itself is abstract matter created by humans. Applicant is mainly using generic/general-purpose computers, processors, and/or computer components/elements/ devices, etc., (for example, “computing device, display in user interface, file, compiling, executable code” (in independent claim 1); “system, processor, memory, display in user interface, file, compiling, executable code” (in independent claim 13); “non-transitory computer-readable storage media, processing device, display in user interface, file, compiling, executable code” (in independent claim 17)) in the amended claims. The terms computationally and compute, as used, do not necessarily mean using a computer or any technical elements as terms “computing” or “to compute” historically predate computers and computing hardware and includes the history of methods intended for pen and paper (or chalk and slate, abacuses, diagram drawn by humans, etc.,) with or without tables/etc. It should be noted that mere automation of a manual process or claiming the improved speed or efficiency inherent with applying the abstract idea on a computer where these purported improvements come solely from the capabilities of a general-purpose computer are not sufficient to transform an abstract idea into a patent-eligible invention. See MPEP 2106.04(a); MPEP 2106.05(a); MPEP 2106.05(f); FairWarning IP, LLC v. Iatric Sys., 120 USPQ2d 1293, 1296 (Fed. Cir. 2016); Credit Acceptance Corp. v. Westlake Services, 123 USPQ2d 1100, 1108-09 (Fed. Cir. 2017); Intellectual Ventures I LLC v. Capital One Bank (USA), 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). The core claimed (with amendments) concept is still an abstract idea as the independent claims (1, 13, 17) and dependent claims (2-11, 13, 15-19, 21-23) still recite receiving/obtaining information/data (where the information itself is abstract in nature – e.g. diagram, workflow, shapes, lines, placement, etc.,), data/information analysis/manipulation (comparing information (e.g. regarding shapes, lines, workflows, etc.,), organizing data/information and determining relationships (between data/information), etc.,) to determine more data/information, possibly obtaining more abstract information/data, and providing this determined data/information for further analysis and reproduction/displaying. The independent claims and dependent claims, under the broadest reasonable interpretation, covers methods of organizing human activity (managing workflows (interaction and relationships between elements of the workflow)) and mental process (observing (diagram and drawings with shapes and lines, etc.,) and evaluating the information regarding human drawn information/diagrams and creating/recreating workflows using generic computing elements using received information).
The independent claims and dependent claims are not integrated into practical application because, as also discussed above, the focus of the claims is simply to use computers and a familiar network as a tool to perform abstract processes (as shown above) involving simple information exchange. Carrying out abstract processes involving information exchange is an abstract idea. See, e.g., BSG, 899 F.3d at 1286; SAP America, 898 F.3d at 1167-68; Affinity Labs of Tex., LLC v. DIRECTV, LLC, 838 F.3d 1253, 1261-62 (Fed. Cir. 2016). Applicant’s additional technical terms/limitations/elements (for example, “computing device, display in user interface, file, compiling, executable code” (in independent claim 1); “system, processor, memory, display in user interface, file, compiling, executable code” (in independent claim 13); “non-transitory computer-readable storage media, processing device, display in user interface, file, compiling, executable code” (in independent claim 17)) are just using generic/general-purpose well-known computing/computer elements – where the actual claimed concepts are abstract and can be done by humans. The generic/general-purpose computers and computing elements/terms/limitations are no more than mere instructions to apply the judicial exception (the above stated abstract idea) in an apply-it fashion using generic/general-purpose computers, processors, and/or computer components/elements/ devices, etc. The additional elements do not integrate the abstract idea in to a practical application because it does not impose any meaningful limits on practicing the abstract idea – i.e. they are just post-solution/extra-solution activities.
Additionally, claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the claims do not recite an improvement to another technology or technical field, an improvement to the functioning of the computer itself, or meaningful limitations beyond generally linking the use of an abstract idea to a particular technological environment. None of the hardware offers a meaningful limitation beyond generally linking the system to a particular technological environment, that is, implementation via computers. Adding generic computer components to perform generic functions that are well‐understood, routine and conventional, such as gathering data, performing calculations, and outputting a result would not transform the claim into eligible subject matter. The additional elements or combination of elements in the independent claims and dependent claims other than the abstract idea per se amounts to no more than: (i) mere instructions to implement the idea on a computer, and/or (ii) recitation of generic computer structure that serves to perform generic computer functions that are well-understood, routine, and conventional activities previously known to the pertinent industry. Applicant is directed to the following citations and references: Digitech Image., LLC v. Electronics for Imaging, Inc.(U.S. Patent No. 6,128,415); and (2) Federal register/Vol. 79, No 241 issued on December 16, 2014, page 74629, column 2, Gottschalk v. Benson. Viewed as a whole, the claims do not purport to improve the functioning of the computer itself, or to improve any other technology or technical field. Use of an unspecified, generic computer does not transform an abstract idea into a patent-eligible invention. Thus, the claim does not amount to significantly more than the abstract idea itself. See Alice Corp. v. CLS Bank Int'l, 110 USPQ2d 1976 (U.S. 2014).
See detailed rejection below.
Prior art discussion:
Applicant’s alleges that Jaeger does not teach or suggest "generating extended lines by computationally lengthening the lines detected in the human-drawn workflow diagram," as recited in amended claim 1. Examiner respectfully disagrees.
Jaeger discloses “joining step includes the step of drawing…line extending (lengthening the line” in Jaeger’s claim 58, “step of connecting includes drawing a line that extends to each of said plurality of recognizable hand drawn entries” in Jaegers claim 79, and additionally in paragraphs 0146-0152 Jaeger states “conversion…drawing is converted…pixels…display…software…PC (computer)…converts…points to a drawing…presented…computer display (shows computationally done).” Jaeger provides a graphic programming based on objects drawn by a user and the creation and utilization of hand drawn objects. Jaeger allows software to "understand" what users are trying to do by what they are drawing. Jaeger, on paragraphs 0174-0175 shows computational lengthening of the line (note that the terms computationally and compute, as used, do not necessarily mean using a computer or any technical elements as terms “computing” or “to compute” historically predate computers and computing hardware and includes the history of methods intended for pen and paper (or chalk and slate, abacuses, diagram drawn by humans, etc.,) with or without tables/etc.) using computer/computing elements by describing how long a line can be drawn. And in paragraph 0210 Jaeger states that there is a recognition of human drawings and that the software redraws the object geometrically (object is identified and machine drawn on screen). Therefore, Jaeger indeed discloses Applicants’ very broad limitation "generating extended lines by computationally lengthening the lines detected in the human-drawn workflow diagram."
Applicant alleges Jaeger thus fails to disclose, teach, or suggest "detecting, using one or more image recognition algorithms, multiple shapes and multiple lines having midpoints that are less than a threshold distance from one another in the human-drawn workflow diagram," as recited in amended claim 13. Examiner respectfully disagrees.
Jaeger discloses, for example, “designed to detect” various shapes (arc, zigzag, etc.,) and “indicates the drawing of a sequence of points that was in the shape of something” in paras. 0206-0208 with paragraph 0215 stating “define and analyze segments…length of each segment is calculated, as well as the end-to-end angle, the start angle, finish angle, the trend of the segment, and the straightness (sustained curvature)…values are used to detect characteristics that are determinants of object shape,” with paragraph 0041 stating “recognize what are various hand drawn images resides in the software code as sets of rules and as real time algorithms” and paragraph 0022 stating “use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects.” Paragraphs 0213-0214 state “a vertex determination is made if the slice angle…exceeds the current threshold angle… past the region…which is at (or close to) the midpoint…places the vertex at the midpoint” with Jager’s claims 56-58 showing “joining…in a utilitarian sequence…joining step includes the step of drawing at least one line extending to and intersecting…in said sequence.” Paragraph 0206-0208 of Jaeger also states, “measure the trend of a single segment…if one drew a zigzag line, the trend of this line (actually a sequence of points) would be very low (close to zero) because each zig trend would tend to cancel out each zag trend.” Therefore, Jaeger indeed discloses Applicant’s broadly claimed limitation of "detecting, using one or more image recognition algorithms, multiple shapes and multiple lines having midpoints that are less than a threshold distance from one another in the human-drawn workflow diagram."
Applicant also alleges Jaeger fails to disclose, teach, or suggest "detecting, using one or more image recognition algorithms, multiple shapes and a line in the human-drawn workflow diagram, the multiple shapes including two or more shapes having centers that are less than a threshold distance from one another," as recited in claim 17. Examiner respectfully disagrees.
Jaeger discloses, for example, “designed to detect” various shapes (arc, zigzag, etc.,) and “indicates the drawing of a sequence of points that was in the shape of something” in paras. 0206-0208 with paragraph 0215 stating “define and analyze segments…length of each segment is calculated, as well as the end-to-end angle, the start angle, finish angle, the trend of the segment, and the straightness (sustained curvature)…values are used to detect characteristics that are determinants of object shape,” with paragraph 0041 stating “recognize what are various hand drawn images resides in the software code as sets of rules and as real time algorithms” and paragraph 0022 stating “use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects.” Paragraphs 0213-0214 state “a vertex determination is made if the slice angle…exceeds the current threshold angle… past the region…which is at (or close to) the midpoint…places the vertex at the midpoint” with Jager’s claims 56-58 showing “joining…in a utilitarian sequence…joining step includes the step of drawing at least one line extending to and intersecting…in said sequence.” Paragraph 0206-0208 of Jaeger also states, “measure the trend of a single segment…if one drew a zigzag line, the trend of this line (actually a sequence of points) would be very low (close to zero) because each zig trend would tend to cancel out each zag trend.” Therefore, Jaeger indeed discloses Applicant’s broadly claimed limitation of "detecting, using one or more image recognition algorithms, multiple shapes and a line in the human-drawn workflow diagram, the multiple shapes including two or more shapes having centers that are less than a threshold distance from one another."
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-11, 13, 15-19, and 21-23 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Note: The above §101 discussion in the Response to Arguments section is fully incorporated in this rejection in addition to the rejection below.
Regarding Step 1 (MPEP 2106.03) of the subject matter eligibility test per MPEP 2106.03, Claims 1-12 are directed to a method (i.e., process), claims 13-16 are directed to a system (i.e. machine), and claims 17-20 are directed to non-transitory computer readable medium (i.e. product or article of manufacture). Accordingly, all claims are directed to one of the four statutory categories of invention.
(Under Step 2) The claimed invention is directed to an abstract idea without significantly more.
(Under Step 2A, Prong 1 (MPEP 2106.04)) The independent claims (1, 13, 17) recite receiving/obtaining information/data (where the information itself is abstract in nature – e.g. diagram, workflow, shapes, lines, placement, etc.,), data/information analysis/manipulation (comparing information (e.g. regarding shapes, lines, workflows, etc.,), organizing data/information and determining relationships (between data/information), etc.,) to determine more data/information, possibly obtaining more abstract information/data, and providing this determined data/information for further analysis and reproduction/displaying. The limitations of the independent claims (1, 13, 17), under the broadest reasonable interpretation, covers methods of organizing human activity (managing workflows (interaction and relationships between elements of the workflow)) and mental process (observing (diagram and drawings with shapes and lines, etc.,) and evaluating the information regarding human drawn information/diagrams and creating/recreating workflows using generic computing elements using received information). If a claims limitation, under its broadest reasonable interpretation, covers the performance of the limitation as fundamental economic principles or practices (including hedging, insurance, mitigating risk); commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations); managing personal behavior or relationships or interactions between people (including scheduling, social activities, teaching, and following rules or instructions), then it falls within the “organizing human activities” grouping of abstract ideas. (MPEP 2106.04). If claim limitations, under its broadest reasonable interpretation, cover the performance of the limitation as concepts performed in the human mind (including an observation, evaluation, judgment, opinion), the claim limitations fall within the Mental process grouping of abstract ideas. (MPEP 2106.04).
Accordingly, since Applicant's claims fall under organizing human activities grouping and mental process grouping, the claims recite an abstract idea.
(Under Step 2A, prong 2 (MPEP 2106.04(d))) This judicial exception is not integrated into a practical application because but for the recitation of well-known generic/general-purpose computing/technology components/elements/terms (for example, “computing device, display in user interface, file, compiling, executable code” (in independent claim 1); “system, processor, memory, display in user interface, file, compiling, executable code” (in independent claim 13); “non-transitory computer-readable storage media, processing device, display in user interface, file, compiling, executable code” (in independent claim 17)), in the context of the independent claims (1, 13, 17), the claims encompass the above stated abstract idea (organizing human activity (managing workflows (interaction and relationships between elements of the workflow)) and mental process (observing (diagram and drawings with shapes and lines, etc.,) and evaluating the information regarding human drawn information/diagrams and creating/recreating workflows using generic computing elements using received information)).
As shown above, the independent claims (1, 13, 17) recite generic/general-purpose computing/technology components/elements/terms/limitations (for example, “computing device, display in user interface, file, compiling, executable code” (in independent claim 1); “system, processor, memory, display in user interface, file, compiling, executable code” (in independent claim 13); “non-transitory computer-readable storage media, processing device, display in user interface, file, compiling, executable code” (in independent claim 17)) which are recited at a high level of generality performing generic/general purpose computer/computing functions. (MPEP 2106.04). The generic/general-purpose computing/technology components/elements/terms/limitations are no more than mere instructions to apply the judicial exception (the above abstract idea – organizing human activity (managing workflows (interaction and relationships between elements of the workflow)) and mental process (observing (diagram and drawings with shapes and lines, etc.,) and evaluating the information regarding human drawn information/diagrams and creating/recreating workflows using generic computing elements using received information)) in an apply-it fashion using generic/general-purpose computing/technology components/elements/terms/limitations (for example, “computing device, display in user interface, file, compiling, executable code” (in independent claim 1); “system, processor, memory, display in user interface, file, compiling, executable code” (in independent claim 13); “non-transitory computer-readable storage media, processing device, display in user interface, file, compiling, executable code” (in independent claim 17)). The CAFC has stated that it is not enough, however, to merely improve abstract processes by invoking a computer merely as a tool. Customedia Techs., LLC v. Dish Network Corp., 951 F.3d 1359, 1364 (Fed. Cir. 2020). The focus of the claims is simply to use computers and a familiar network as a tool to perform abstract processes (discussed above) involving simple information exchange. Carrying out abstract processes involving information exchange is an abstract idea. See, e.g., BSG, 899 F.3d at 1286; SAP America, 898 F.3d at 1167-68; Affinity Labs of Tex., LLC v. DIRECTV, LLC, 838 F.3d 1253, 1261-62 (Fed. Cir. 2016). And use of standard computers and networks to carry out those functions—more speedily, more efficiently, more reliably—does not make the claims any less directed to that abstract idea. See Alice Corp., 573 U.S. at 222-25; Customedia, 951 F.3d at 1364; Trading Techs. Int'l, Inc. v. IBG LLC, 921 F.3d 1084, 1092-93 (Fed. Cir. 2019); SAP America, 898 F.3d at 1167; Intellectual Ventures I LLC v. Symantec Corp., 838 F.3d 1307, 1314 (Fed. Cir. 2016); Electric Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1353, 1355 (Fed. Cir. 2016); Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 1370 (Fed. Cir. 2015); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355 (Fed. Cir. 2014). Accordingly, the additional elements (for example, “computing device, display in user interface, file, compiling, executable code” (in independent claim 1); “system, processor, memory, display in user interface, file, compiling, executable code” (in independent claim 13); “non-transitory computer-readable storage media, processing device, display in user interface, file, compiling, executable code” (in independent claim 17)) do not integrate the abstract idea in to a practical application because it does not impose any meaningful limits on practicing the abstract idea – i.e. they are just post-solution/extra-solution activities.
(Under Step 2B (MPEP 2106.05)) The independent claims (1, 13, 17) do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the claims do not recite an improvement to another technology or technical field, an improvement to the functioning of the computer itself, or meaningful limitations beyond generally linking the use of an abstract idea to a particular technological environment. The independent claims recite using known generic/general-purpose computing/technology components/elements/terms/limitations (for example, “computing device, display in user interface, file, compiling, executable code” (in independent claim 1); “system, processor, memory, display in user interface, file, compiling, executable code” (in independent claim 13); “non-transitory computer-readable storage media, processing device, display in user interface, file, compiling, executable code” (in independent claim 17)). For the role of a computer in a computer implemented invention to be deemed meaningful in the context of this analysis, it must involve more than performance of "well-understood, routine, [and] conventional activities previously known to the industry." Alice Corp. v. CLS Bank Int'l, 110 USPQ2d 1976 (U.S. 2014), at 2359 (quoting Mayo, 132 S. Ct. at 1294 (internal quotation marks and brackets omitted)). These activities as claimed by the Applicant are all well-known and routine tasks in the field of art – as can been seen in the specification of Applicant’s application (for example, see Applicant’s specification at, for example, figure 4 and Pages 24-25 [where Applicant recites general-purpose/generic computers/processors/etc., and generic/general-purpose computing components/devices/etc., in Applicant’s specification]) and/or the specification of the below cited art (used in the rejection below and on the PTO-892) and/or also as noted in the court cases in §2106.05 in the MPEP. Further, "the mere recitation of a generic computer cannot transform a patent ineligible abstract idea into a patent-eligible invention." Alice at 2358. None of the hardware offers a meaningful limitation beyond generally linking the system to a particular technological environment, that is, implementation via computers. Adding generic computer components to perform generic functions that are well‐understood, routine and conventional, such as gathering data, performing calculations, and outputting a result would not transform the claims into eligible subject matter. Abstract ideas are excluded from patent eligibility based on a concern that monopolization of the basic tools of scientific and technological work might impede innovation more than it would promote it. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the claims require no more than a generic computer to perform generic computer functions. The additional elements (for example, “computing device, display in user interface, file, compiling, executable code” (in independent claim 1); “system, processor, memory, display in user interface, file, compiling, executable code” (in independent claim 13); “non-transitory computer-readable storage media, processing device, display in user interface, file, compiling, executable code” (in independent claim 17)) or combination of elements in the claims other than the abstract idea per se amounts to no more than: (i) mere instructions to implement the idea on a computer, and/or (ii) recitation of generic computer structure that serves to perform generic computer functions that are well-understood, routine, and conventional activities previously known to the pertinent industry. Applicant is directed to the following citations and references: Digitech Image., LLC v. Electronics for Imaging, Inc. (758 F.3d 1344 (2014) discussing U.S. Patent No. 6,128,415); and (2) Federal register/Vol. 79, No 241 issued on December 16, 2014, page 74629, column 2, Gottschalk v. Benson. Viewed as a whole, the independent claims do not purport to improve the functioning of the computer itself, or to improve any other technology or technical field. Use of an unspecified, generic computer does not transform an abstract idea into a patent-eligible invention. Thus, the independent claims do not amount to significantly more than the abstract idea itself. See Alice Corp. v. CLS Bank Int'l, 110 USPQ2d 1976 (U.S. 2014).
The dependent claims (2-11, 13, 15-19, 21-23) further define the independent claims and merely narrow the described abstract idea, but not adding significantly more than the abstract idea. The dependent claims either individually or in combination are merely an extension of the abstract idea itself. The above rejection discussed for the independent claims fully applies to the dependent claims.
The dependent claims (2-11, 13, 15-19, 21-23) further state using obtained data/information (where the information itself is abstract in nature – e.g. diagram, workflow, shapes, lines, placement, etc.,), data/information analysis/manipulation (comparing information (e.g. regarding shapes, lines, workflows, etc.,), organizing data/information and determining relationships (between data/information), etc.,) to determine more data/information, possibly obtaining more abstract information/data, and providing this determined data/information for further analysis and reproduction/displaying. These dependent claims also cover methods of organizing human activity (managing workflows (interaction and relationships between elements of the workflow)) and mental process (observing (diagram and drawings with shapes and lines, etc.,) and evaluating the information regarding human drawn information/diagrams and creating/recreating workflows using generic computing elements using received information).
This judicial exception is not integrated into a practical application because the claims and specification recite additional elements as generic/general-purpose computing/technology components/elements/terms/limitations (convolutional neural network (term only stated but no technical details provided on how it functions as per Applicant’s claims), trained/training (only stated but no technical details provided on how it functions as per Applicant’s claims), optical character recognition, file, executable code (software), memory, processing (processor), graphical programming, (in claim 1’s dependent claims 2-11, 21-23); system (in claim 13’s dependent claims 14-16); non-transitory computer-readable storage media, file, executable code (software) (in claim 17’s dependent claims 18-19)) performing generic computer/computing/technology functions. (MPEP 2106.04). The dependent claims merely use the same general technological environment and instructions as the independent claims above to implement the abstract idea. The generic/general-purpose computing/technology components/elements/terms/limitations are no more than mere instructions to apply the judicial exception (the above abstract idea – organizing human activity (managing workflows (interaction and relationships between elements of the workflow)) and mental process (observing (diagram and drawings with shapes and lines, etc.,) and evaluating the information regarding human drawn information/diagrams and creating/recreating workflows using generic computing elements using received information)) in an apply-it fashion using generic/general-purpose computing/technology components/elements/terms/limitations (convolutional neural network (term only stated but no technical details provided on how it functions as per Applicant’s claims), trained/training (only stated but no technical details provided on how it functions as per Applicant’s claims), optical character recognition, file, executable code (software), memory, processing (processor), graphical programming, (in claim 1’s dependent claims 2-11, 21-23); system (in claim 13’s dependent claims 14-16); non-transitory computer-readable storage media, file, executable code (software) (in claim 17’s dependent claims 18-19)). Hence, the additional elements (convolutional neural network (term only stated but no technical details provided on how it functions as per Applicant’s claims), trained/training (only stated but no technical details provided on how it functions as per Applicant’s claims), optical character recognition, file, executable code (software), memory, processing (processor), graphical programming, (in claim 1’s dependent claims 2-11, 21-23); system (in claim 13’s dependent claims 14-16); non-transitory computer-readable storage media, file, executable code (software) (in claim 17’s dependent claims 18-19)) do not integrate the abstract idea in to a practical application because they does not impose any meaningful limits on practicing the abstract idea – i.e. they are just post-solution/extra-solution activities.
Also, the dependent claims (2-11, 13, 15-19, 21-23) either individually or in combination are merely an extension of the abstract idea itself and the dependent claims (similar to the independent claims) do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the claims require no more than a generic computer to perform generic computer functions. The additional elements (convolutional neural network (term only stated but no technical details provided on how it functions as per Applicant’s claims), trained/training (only stated but no technical details provided on how it functions as per Applicant’s claims), optical character recognition, file, executable code (software), memory, processing (processor), graphical programming, (in claim 1’s dependent claims 2-11, 21-23); system (in claim 13’s dependent claims 14-16); non-transitory computer-readable storage media, file, executable code (software) (in claim 17’s dependent claims 18-19)) or combination of elements in the dependent claims other than the abstract idea per se amounts to no more than: (i) mere instructions to implement the idea on a computer, and/or (ii) recitation of generic computer structure that serves to perform generic computer functions that are well-understood, routine, and conventional activities previously known to the pertinent industry. Applicant is directed to the following citations and references: Digitech Image., LLC v. Electronics for Imaging, Inc. (758 F.3d 1344 (2014) discussing U.S. Patent No. 6,128,415); and (2) Federal register/Vol. 79, No 241 issued on December 16, 2014, page 74629, column 2, Gottschalk v. Benson. Viewed as a whole, dependent claims do not purport to improve the functioning of the computer itself, or to improve any other technology or technical field. Use of an unspecified, generic computer does not transform an abstract idea into a patent-eligible invention. Thus, the dependent claims do not amount to significantly more than the abstract idea itself. See Alice Corp. v. CLS Bank Int'l, 110 USPQ2d 1976 (U.S. 2014).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2 and 5-11, 13, 15-19, and 21-23 are rejected under 35 U.S.C. 103 as being unpatentable over Jaeger et al., (US 2002/0141643) in view of Dillner (US 2004/0090439).
As per claim 1, Jaeger discloses a method implemented by at least one computing device, the method comprising:
receiving a digital image of a human-drawn workflow diagram (¶¶ 0006 [recognize hand drawn objects…inputting data on…screen; with 0142 [converts… graphic objects…digital…processing]], 0041 [hand drawn images are recognized…input device]; claim 51 [input device to freehand draw…hand drawn entry into a functional device]);
detecting, using one or more image recognition algorithms, shapes and lines in the human-drawn workflow diagram (¶¶ 0206-0208 [detecting…line…shape; with 0041 [recognize what are various hand drawn images resides in the software code as sets of rules and as real time algorithms; with 0022 [use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects ], 0055 [hand drawn…flow diagram]]; Jaeger’s claims 56-58 [showing workflow – joining…in a utilitarian sequence…joining step includes the step of drawing at least one line extending to and intersecting…in said sequence]);
generating extended lines by computationally lengthening the lines detected in the human-drawn workflow diagram (Jaeger’s claim 58 [joining step includes the step of drawing…line extending (lengthening the line); see with ¶¶ 0146-0152 [conversion…drawing is converted…pixels…display…software…PC (computer)…converts…points to a drawing…presented…computer display (shows computationally done)], 0174-0175 [(shows computationally lengthening of the line using computer/computing elements)]], claim 79 [drawing line that extends (lengthening lines)]; also see ¶¶ 0210-0213, 0228, 0403);
determining relationships between the shapes based on relative positionings of the extended lines with respect to the shapes (see citations above and also see ¶¶ 0019 [context…location, proximity], 0024 [rectangle (shape)…location…can affect other objects…drawing connecting lines; see with 0054 and 0063-0064 [arrows/lines…can be hand drawn to designate links (relationships) between any graphic object or group of objects (shapes) and any other graphic object or group of objects; see with 0065-0067 [example given – arrow/line…nature of the link]]]; see also 0320-0323);
generating, for display in a user interface, a structured workflow diagram including compute nodes representing the shapes that are connected by generated lines representing the relationships (¶¶ 0064 [the definition(s) of arrows' functions can be determined by the graphical context of the arrows: the way the arrows are drawn, where the arrows are drawn, the span of the graphical context of the arrows, where the arrows start and where they end up, and as mentioned above, when they were drawn or when they were started and then completed. It is the graphical context which the arrow encompasses together with the user defined nature of an arrow (herein termed "arrow logics") that can define that arrow's function (workflow is described here). When one draws an arrow between certain types of graphical objects (nodes), different operations may be commanded, depending upon the context(s) of the arrow; see with 0065-0068 [example of this workflow – object recognized by this system may have its own info window that can be used for the user-specification…definitions, assignment(s), operations, functions, actions…one can link these two knobs by drawing an arrow between them. Note that the direction of the arrow between the knobs is crucial in defining the nature of the link…operation of the arrow…arrows generally represent actions that are governed by logics and as such, once an arrow is drawn and recognized and its action carried out (workflow)], 0056-0058 [user interface… graphic on a screen], 0024 [display on screen/interface]]); and
generating a structured workflow representing the structured workflow diagram (¶¶ 0038 [hand drawn graphics…used to create…path…flow…order of events, types of devices, groupings of devices, relationships between devices; with 0055+ [hand drawn…becomes…flow diagram]]);
compiling the structured workflow file into executable code (see citations above for claim 1 and see ¶¶ 0002 [computer…programmed…software; with 0014 [functions to be carried out by the software (code)], 0041 [software code as sets of rules and as real time algorithms; with 0072 [configurable hardware, processes, actions, functions and the like, with software being implemented]], 0147 [operating system of the machine that this software is running on, i.e., a PC]); and
running the executable code to perform a workflow defined by the human- drawn workflow diagram (see citations above and see with, for example, ¶¶ 0072 [hand drawing of objects which can then be turned into virtual or actual…processes (workflow), actions (workflows), functions and the like, with software being implemented…processes, actions, functions, etc. being created, controlled and/or initiated; with 0038 [hand drawn graphics…used to create…path…flow…order of events, types of devices, groupings of devices, relationships between devices], 0014 [functions to be carried out by the software (code)]]).
Jaeger does not explicitly state file.
Analogous art Dillner discloses files and flow diagrams (¶¶ 0063 [hand drawn…results in flow diagram; with 0075 [transformed to a file that specifies…diagram; with 0209 [diagrams usually result in the construction of a directed graph that appropriately represents the flow of data in the diagram]], 0192 [representation…visual…environment…converted into a program understood by the visual programming environment (such as a text file specifying a diagram)]]).
Therefore, it would be obvious to one of ordinary skill in the art to include in the system/method of Jaeger file as taught by analogous art Dillner in order to efficiently provide essential instructions to the computing environment since doing so could be performed readily by any person of ordinary skill in the art, with neither undue experimentation, nor risk of unexpected results (KSR-G/TSM); and also since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Dillner would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141).
As per claim 2, Jaeger discloses the method of claim 1, wherein detecting the shapes and the lines includes extracting coordinates of the shapes and the lines within the image, and determining the relationships is based on the coordinates ¶¶ 0006 [location of drawing], 0019 [context…location, proximity], 0024[rectangle (shape)…location…can affect other objects…drawing connecting lines; see with 0053-0054 [location…image…exact location of…image] and 0063-0064 [arrows/lines…can be hand drawn to designate links (relationships) between any graphic object or group of objects (shapes) and any other graphic object or group of objects; see with 0065-0067 [example given – arrow/line…nature of the link]]]; see also 0320-0323).
As per claim 5, Jaeger discloses the method of claim 1, wherein detecting the lines includes detecting multiple lines having midpoints that are less than a threshold distance from one another (¶¶ 0206-0208 [detecting…line…shape; with 0041 [recognize what are various hand drawn images resides in the software code as sets of rules and as real time algorithms; with 0022 [use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects ], 0055 [hand drawn…flow diagram], and 0214 [a vertex determination is made if the slice angle…exceeds the current threshold angle… past the region…which is at (or close to) the midpoint…places the vertex at the midpoint]]; Jaeger’s claims 56-58 [showing workflow – joining…in a utilitarian sequence…joining step includes the step of drawing at least one line extending to and intersecting…in said sequence]) and
retaining a single line of the multiple lines, wherein determining the relationships includes determining a relationship between two shapes based on the single line (¶¶ 0064 [the definition(s) of arrows' functions can be determined by the graphical context of the arrows: the way the arrows are drawn, where the arrows are drawn, the span of the graphical context of the arrows, where the arrows start and where they end up, and as mentioned above, when they were drawn or when they were started and then completed. It is the graphical context which the arrow encompasses together with the user defined nature of an arrow (herein termed "arrow logics") that can define that arrow's function (workflow is described here). When one draws an arrow between certain types of graphical objects (nodes), different operations may be commanded, depending upon the context(s) of the arrow; see with 0139 [enabling a single…graphic/line to represent and implement], 0065-0068 [example of this workflow – object recognized by this system may have its own info window that can be used for the user-specification…definitions, assignment(s), operations, functions, actions…one can link these two knobs by drawing an arrow between them…the direction of the arrow between the knobs is crucial in defining the nature of the link…operation of the arrow…arrows generally represent actions that are governed by logics and as such, once an arrow is drawn and recognized and its action carried out (workflow)]]).
As per claim 18, claim 18 discloses substantially similar limitations as claim 5 above; and therefore claim 18 is rejected under the same rationale and reasoning as presented above for claim 5.
As per claim 6, Jaeger discloses the method of claim 1, further comprising removing at least one shape detected in the human-drawn workflow diagram based on a degree of difference between a size of the at least one shape relative to an average size of the shapes in the human-drawn workflow diagram, wherein determining the relationships is based on the relative positionings of the lines with respect to the shapes excluding the at least one shape (¶¶ 0063 [lines/shapes…not required; with 0330 [editing…chop tool…cuts…line/lines], 0155 [objects…eliminated; with 0210 [not allowed…size…eliminated], 0249 [size limits…eliminate…object…size differential may be employed to distinguish the two objects]; 0245-0247 [elimination of…shapes], 0139 [enabling a single…graphic/line to represent and implement], 0024 [rectangle (shape)…location…can affect other objects…drawing connecting lines; see with 0054 and 0063-0064 [arrows/lines…can be hand drawn to designate links (relationships) between any graphic object or group of objects (shapes) and any other graphic object or group of objects; see with 0065-0067 [example given – arrow/line…nature of the link]]]; see also 0320-0323).
As per claim 14, claim 14 discloses substantially similar limitations as claim 6 above; and therefore claim 14 is rejected under the same rationale and reasoning as presented above for claim 6.
As per claim 7, Jaeger discloses the method of claim 1, wherein detecting the shapes includes detecting multiple shapes having centers that are within a threshold distance from one another and generating a merged shape by merging the multiple shapes, wherein determining the relationships includes determining a relationship between the merged shape and an additional shape (¶¶ 0206-0208 [detecting…line…shape; with 0212-0214 [determine…distance…threshold value…does not exceed the threshold…midpoint], 0041 [recognize what are various hand drawn images resides in the software code as sets of rules and as real time algorithms; with 0022 [use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects], and 0214 [a vertex determination is made if the slice angle…exceeds the current threshold angle… past the region…which is at (or close to) the midpoint…places the vertex at the midpoint]]; Jaeger’s claims 56-58 [showing workflow – joining…in a utilitarian sequence…joining step includes the step of drawing at least one line extending to and intersecting…in said sequence], se with 0091 [combination of two or more…shapes; with 0247 [example – although a rectangle or folder or triangle…combined…identification as a circle], 0328 [operates to combine other graphic objects]], 0019 [context…location, proximity], 0024 [rectangle (shape)…location…can affect other objects…drawing connecting lines; see with 0054 and 0063-0064 [arrows/lines…can be hand drawn to designate links (relationships) between any graphic object or group of objects (shapes) and any other graphic object or group of objects; see with 0065-0067 [example given – arrow/line…nature of the link]]]; see also 0320-0323).
As per claim 8, Jaeger discloses the method of claim 1, wherein detecting the lines includes detecting arrows and directions in which the arrows are pointed, wherein determining the relationships is further based on the directions (see citations above for claim 1 and see ¶¶ 0022 [use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects; with 0054 [arrow…starting and ending points], 0062-0066 [arrow logics…direction of the arrow…arrowhead is pointing]]).
As per claim 9, Jaeger discloses the method of claim 1, further comprising: detecting, using an algorithm, characters in the human-drawn workflow diagram; assigning the characters to the shapes and the lines based on relative positionings of the characters with respect to the shapes and the lines, respectively (see citations above for claim 1 and see ¶¶ 0206-0208 [detecting…line…shape; with 0041 [recognize what are various hand drawn images resides in the software code as sets of rules and as real time algorithms; with 0022-0025 [use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects…assign colors…color defines function and/or assignment and/or properties and/or actions], 0055 [hand drawn…flow diagram]], 0064 [the definitions of arrows' functions can be determined by the graphical context of the arrows: the way the arrows are drawn, where the arrows are drawn, the span of the graphical context of the arrows, where the arrows start and where they end up, and as mentioned above, when they were drawn or when they were started and then completed. It is the graphical context which the arrow encompasses together with the user defined nature of an arrow (herein termed "arrow logics") that can define that arrow's function (workflow is described here). When one draws an arrow between certain types of graphical objects, different operations may be commanded, depending upon the context(s) of the arrow; see with 0065-0068 [example of this workflow – object recognized by this system may have its own info window that can be used for the user-specification…definitions, assignments, operations, functions, actions…one can link these two knobs by drawing an arrow between them. Note that the direction of the arrow between the knobs is crucial in defining the nature of the link…operation of the arrow…arrows generally represent actions that are governed by logics and as such, once an arrow is drawn and recognized and its action carried out], 0056-0058 [user interface… graphic on a screen], 0024 [display on screen/interface]]);
associating the compute nodes with operation types corresponding to the characters assigned to the shapes; and associating the relationships with relationship types corresponding to the characters assigned to the lines (see citations above for claim 1 and see ¶¶ 0064 [the definitions of arrows' functions can be determined by the graphical context of the arrows: the way the arrows are drawn, where the arrows are drawn, the span of the graphical context of the arrows, where the arrows start and where they end up, and as mentioned above, when they were drawn or when they were started and then completed. It is the graphical context which the arrow encompasses together with the user defined nature of an arrow (herein termed "arrow logics") that can define that arrow's function (workflow is described here). When one draws an arrow between certain types of graphical objects (nodes), different operations may be commanded, depending upon the context(s) of the arrow; see with 0065-0068 [example of this workflow – object recognized by this system may have its own info window that can be used for the user-specification…definitions, assignments, operations, functions, actions…one can link these two knobs by drawing an arrow between them. Note that the direction of the arrow between the knobs is crucial in defining the nature of the link…operation of the arrow…arrows generally represent actions that are governed by logics and as such, once an arrow is drawn and recognized and its action carried out (workflow)]).
Jaeger does not state optical character recognition.
Analogous art Dillner discloses optical character recognition (¶¶ 0261-0262 [optical character recognition…OCR]).
Therefore, it would be obvious to one of ordinary skill in the art to include in the system/method of Jaeger optical character recognition as taught by analogous art Dillner in order to efficiently process images (machine vision tasks) since doing so could be performed readily by any person of ordinary skill in the art, with neither undue experimentation, nor risk of unexpected results (KSR-G/TSM); and also since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Dillner (optical character recognition (OCR) is old and well-known in image processing) would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141).
As per claim 10, Jaeger discloses the method of claim 1, further comprising: detecting, using an algorithm, characters in the human-drawn workflow diagram; assigning the characters to the shapes and the lines based on relative positionings of the characters with respect to the shapes and the lines, respectively (see citations above for claim 1 and see ¶¶ 0206-0208 [detecting…line…shape; with 0041 [recognize what are various hand drawn images resides in the software code as sets of rules and as real time algorithms; with 0022-0025 [use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects…assign colors…color defines function and/or assignment and/or properties and/or actions], 0055 [hand drawn…flow diagram]], 0064 [the definitions of arrows' functions can be determined by the graphical context of the arrows: the way the arrows are drawn, where the arrows are drawn, the span of the graphical context of the arrows, where the arrows start and where they end up, and as mentioned above, when they were drawn or when they were started and then completed. It is the graphical context which the arrow encompasses together with the user defined nature of an arrow (herein termed "arrow logics") that can define that arrow's function (workflow is described here). When one draws an arrow between certain types of graphical objects, different operations may be commanded, depending upon the context(s) of the arrow; see with 0065-0068 [example of this workflow – object recognized by this system may have its own info window that can be used for the user-specification…definitions, assignments, operations, functions, actions…one can link these two knobs by drawing an arrow between them. Note that the direction of the arrow between the knobs is crucial in defining the nature of the link…operation of the arrow…arrows generally represent actions that are governed by logics and as such, once an arrow is drawn and recognized and its action carried out], 0056-0058 [user interface… graphic on a screen], 0024 [display on screen/interface]]); and
labeling the compute nodes and the generated lines with identifiers in the structured workflow diagram based on the characters assigned to the shapes and the lines, respectively (see citations above for claim 1 and see ¶¶ 0064 [the definitions of arrows' functions can be determined by the graphical context of the arrows: the way the arrows are drawn, where the arrows are drawn, the span of the graphical context of the arrows, where the arrows start and where they end up, and as mentioned above, when they were drawn or when they were started and then completed. It is the graphical context which the arrow encompasses together with the user defined nature of an arrow (herein termed "arrow logics") that can define that arrow's function (workflow is described here). When one draws an arrow between certain types of graphical objects (nodes), different operations may be commanded, depending upon the context(s) of the arrow; see with 0065-0068 [example of this workflow – object recognized by this system may have its own info window that can be used for the user-specification…definitions, assignments, operations, functions, actions…one can link these two knobs by drawing an arrow between them. Note that the direction of the arrow between the knobs is crucial in defining the nature of the link…operation of the arrow…arrows generally represent actions that are governed by logics and as such, once an arrow is drawn and recognized and its action carried out (workflow)]).
Jaeger does not state optical character recognition.
Analogous art Dillner discloses optical character recognition (¶¶ 0261-0262 [optical character recognition…OCR]).
Therefore, it would be obvious to one of ordinary skill in the art to include in the system/method of Jaeger optical character recognition as taught by analogous art Dillner in order to efficiently process images (machine vision tasks) since doing so could be performed readily by any person of ordinary skill in the art, with neither undue experimentation, nor risk of unexpected results (KSR-G/TSM); and also since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Dillner (optical character recognition (OCR) is old and well-known in image processing) would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141).
As per claim 11, Jaeger discloses the method of claim 1, wherein detecting the shapes includes detecting different shape types in the human-drawn workflow diagram (see citations for claim 1 above and see, for example, ¶¶ 0089 [rectangles, squares, equilateral triangles, etc.,], 0200 [circle or a rectangle or triangle], 0218 [hand drawn…type of…shape…rectangle…triangle…folder]),
wherein generating the structured workflow includes associating the compute nodes in the structured workflow with different operation types corresponding to the different shape types (see citations for claim 1 above and see, for example, ¶¶ 0064 [the definitions of arrows' functions can be determined by the graphical context of the arrows: the way the arrows are drawn, where the arrows are drawn, the span of the graphical context of the arrows, where the arrows start and where they end up, and as mentioned above, when they were drawn or when they were started and then completed. It is the graphical context which the arrow encompasses together with the user defined nature of an arrow (herein termed "arrow logics") that can define that arrow's function (workflow is described here). When one draws an arrow between certain types of graphical objects (nodes), different operations may be commanded, depending upon the context(s) of the arrow; see with 0065-0068 [example of this workflow – object recognized by this system may have its own info window that can be used for the user-specification…definitions, assignments, operations, functions, actions…one can link these two knobs by drawing an arrow between them. Note that the direction of the arrow between the knobs is crucial in defining the nature of the link…operation of the arrow…arrows generally represent actions that are governed by logics and as such, once an arrow is drawn and recognized and its action carried out (workflow)]).
Jaeger does not explicitly state file.
Analogous art Dillner discloses files and flow diagrams (¶¶ 0063 [hand drawn…results in flow diagram; with 0075 [transformed to a file that specifies…diagram; with 0209 [diagrams usually result in the construction of a directed graph that appropriately represents the flow of data in the diagram]], 0192 [representation…visual…environment…converted into a program understood by the visual programming environment (such as a text file specifying a diagram)]]).
Therefore, it would be obvious to one of ordinary skill in the art to include in the system/method of Jaeger file as taught by analogous art Dillner in order to efficiently provide essential instructions to the computing environment since doing so could be performed readily by any person of ordinary skill in the art, with neither undue experimentation, nor risk of unexpected results (KSR-G/TSM); and also since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Dillner would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141).
As per claim 13, Jaeger discloses system, comprising: at least one processor; and storing instructions, which when executed by the at least one processor, cause the at least one processor to perform operations (¶¶ 0004 [system], 0028 [processor…hardware…software], 0093 [system…processor], 0184, 0212-0213 [system stores data], 0230-0235, 0325) including:
receiving a digital image of a human-drawn workflow diagram (¶¶ 0006 [recognize hand drawn objects…inputting data on…screen; with 0142 [converts… graphic objects…digital…processing]], 0041 [hand drawn images are recognized…input device]; claim 51 [input device to freehand draw…hand drawn entry into a functional device]);
detecting, using one or more image recognition algorithms, multiple shapes and multiple lines having midpoints that are less than a threshold distance from one another in the human-drawn workflow diagram (¶¶ 0206-0208 [detecting…line…shape; with 0041 [recognize what are various hand drawn images resides in the software code as sets of rules and as real time algorithms; with 0022 [use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects ], 0055 [hand drawn…flow diagram], 0214 [a vertex determination is made if the slice angle…exceeds the current threshold angle… past the region…which is at (or close to) the midpoint…places the vertex at the midpoint], 0215 [define and analyze segments…the length of each segment is calculated, as well as the end-to-end angle, the start angle, finish angle, the trend of the segment, and the straightness (sustained curvature)…values are used to detect characteristics that are determinants of object shape]]; Jaeger’s claims 56-58 [showing workflow – joining…in a utilitarian sequence…joining step includes the step of drawing at least one line extending to and intersecting…in said sequence]);
retaining a single line of the multiple lines by removing one or more lines of the multiple lines having the midpoints that are less than the threshold distance from one another (¶¶ 0063 [lines…not required; with 0330 [editing…chop tool…cuts…line/lines], 0139 [enabling a single…graphic/line to represent and implement]; and also 0206-0208 [measure the trend of a single segment…if one drew a zigzag line, the trend of this line (actually a sequence of points) would be very low (close to zero) because each zig trend would tend to cancel out each zag trend], 0332-0333 [line of the X graphic…delete], with 0055 [hand drawn…flow diagram], and 0214 [a vertex determination is made if the slice angle…exceeds the current threshold angle… past the region…which is at (or close to) the midpoint…places the vertex at the midpoint]]; Jaeger’s claims 56-58 [showing workflow – joining…in a utilitarian sequence…joining step includes the step of drawing at least one line extending to and intersecting…in said sequence]]);
determining a relationship between the multiple shapes based on a relative positioning of the single line with respect to the multiple shapes (see citations above and also see ¶¶ 0019 [context…location, proximity], 0024[rectangle (shape)…location…can affect other objects…drawing connecting lines; see with 0054 and 0063-0064 [arrows/lines…can be hand drawn to designate links (relationships) between any graphic object or group of objects (shapes) and any other graphic object or group of objects; see with 0065-0067 [example given – arrow/line…nature of the link]]]; see also 0320-0323);
generating, for display in a user interface, a structured workflow diagram including compute nodes representing the multiple shapes that are connected by a generated line representing the relationship (¶¶ 0064 [the definition(s) of arrows' functions can be determined by the graphical context of the arrows: the way the arrows are drawn, where the arrows are drawn, the span of the graphical context of the arrows, where the arrows start and where they end up, and as mentioned above, when they were drawn or when they were started and then completed. It is the graphical context which the arrow encompasses together with the user defined nature of an arrow (herein termed "arrow logics") that can define that arrow's function (workflow is described here). When one draws an arrow between certain types of graphical objects (nodes), different operations may be commanded, depending upon the context(s) of the arrow; see with 0065-0068 [example of this workflow – object recognized by this system may have its own info window that can be used for the user-specification…definitions, assignment(s), operations, functions, actions…one can link these two knobs by drawing an arrow between them. Note that the direction of the arrow between the knobs is crucial in defining the nature of the link…operation of the arrow…arrows generally represent actions that are governed by logics and as such, once an arrow is drawn and recognized and its action carried out (workflow)], 0056-0058 [user interface… graphic on a screen], 0024 [display on screen/interface]]); and
generating a structured workflow representing the structured workflow diagram (¶¶ 0038 [hand drawn graphics…used to create…path…flow…order of events, types of devices, groupings of devices, relationships between devices; with 0055+ [hand drawn…becomes…flow diagram]]);
compiling the structured workflow file into executable code (see citations above for claim 1 and see ¶¶ 0002 [computer…programmed…software; with 0014 [functions to be carried out by the software (code)], 0041 [software code as sets of rules and as real time algorithms; with 0072 [configurable hardware, processes, actions, functions and the like, with software being implemented]], 0147 [operating system of the machine that this software is running on, i.e., a PC]); and
running the executable code to perform a workflow defined by the human- drawn workflow diagram (see citations above and see with, for example, ¶¶ 0072 [hand drawing of objects which can then be turned into virtual or actual…processes (workflow), actions (workflows), functions and the like, with software being implemented…processes, actions, functions, etc. being created, controlled and/or initiated; with 0038 [hand drawn graphics…used to create…path…flow…order of events, types of devices, groupings of devices, relationships between devices], 0014 [functions to be carried out by the software (code)]]).
Jaeger does not explicitly state memory (although Jaeger does disclose computing system storing information).
Analogous art Dillner discloses memory (¶¶ 0072 [stored in computer memory], 0081 [computer memory]).
Therefore, it would be obvious to one of ordinary skill in the art to include in the system/method of Jaeger memory as taught by analogous art Dillner since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Dillner would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141).
Jaeger does not explicitly state file.
Analogous art Dillner discloses files and flow diagrams (¶¶ 0063 [hand drawn…results in flow diagram; with 0075 [transformed to a file that specifies…diagram; with 0209 [diagrams usually result in the construction of a directed graph that appropriately represents the flow of data in the diagram]], 0192 [representation…visual…environment…converted into a program understood by the visual programming environment (such as a text file specifying a diagram)]]).
Therefore, it would be obvious to one of ordinary skill in the art to include in the system/method of Jaeger file as taught by analogous art Dillner in order to efficiently provide essential instructions to the computing environment since doing so could be performed readily by any person of ordinary skill in the art, with neither undue experimentation, nor risk of unexpected results (KSR-G/TSM); and also since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Dillner would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141).
As per claim 15, Jaeger discloses the system of claim 13, wherein detecting the multiple shapes includes detecting two or more shapes having centers that are within a threshold distance from one another and generating a merged shape by merging the two or more shapes, wherein determining the relationship includes determining the relationship between the merged shape and an additional shape of the multiple shapes (¶¶ 0206-0208 [detecting…line…shape; with 0212-0214 [determine…distance…threshold value…does not exceed the threshold…midpoint], 0041 [recognize what are various hand drawn images resides in the software code as sets of rules and as real time algorithms; with 0022 [use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects], and 0214 [a vertex determination is made if the slice angle…exceeds the current threshold angle… past the region…which is at (or close to) the midpoint…places the vertex at the midpoint]]; Jaeger’s claims 56-58 [showing workflow – joining…in a utilitarian sequence…joining step includes the step of drawing at least one line extending to and intersecting…in said sequence], see with 0091 [combination of two or more…shapes; with 0247 [example – although a rectangle or folder or triangle…combined…identification as a circle], 0328 [operates to combine other graphic objects]], 0019 [context…location, proximity], 0024 [rectangle (shape)…location…can affect other objects…drawing connecting lines; see with 0054 and 0063-0064 [arrows/lines…can be hand drawn to designate links (relationships) between any graphic object or group of objects (shapes) and any other graphic object or group of objects; see with 0065-0067 [example given – arrow/line…nature of the link]]]; see also 0320-0323).
As per claim 16, Jaeger discloses the system of claim 13, the operations further comprising extending the single line, resulting in an extended line (Jaeger’s claim 58 [joining step includes the step of drawing at least one line extending], claim 79 [line that extends]; also see ¶¶ 0403),
wherein determining the relationship is based on the relative positioning of the extended line with respect to the multiple shapes (see citations above and also see ¶¶ 0019 [context…location, proximity], 0024[rectangle (shape)…location…can affect other objects…drawing connecting lines; see with 0054 and 0063-0064 [arrows/lines…can be hand drawn to designate links (relationships) between any graphic object or group of objects (shapes) and any other graphic object or group of objects; see with 0065-0067 [example given – arrow/line…nature of the link]]]; see also 0320-0323).
As per claim 19, claim 19 discloses substantially similar limitations as claim 16 above; and therefore claim 19 is rejected under the same rationale and reasoning as presented above for claim 16.
As per claim 17, Jaeger discloses instructions that, responsive to execution by at least one processing device, cause the at least one processing device to perform operations (¶¶ 0028 [processor…hardware…software], 0093 [system…processor], 0184, 0212-0213 [system stores data], 0230-0235, 0325) including:
receiving an image of a human-drawn workflow diagram (¶¶ 0006 [recognize hand drawn objects…inputting data on…screen], 0041 [hand drawn images are recognized…input device]; claim 51 [input device to freehand draw…hand drawn entry into a functional device]);
detecting, using one or more image recognition algorithms, multiple shapes and a line in the human-drawn workflow diagram, the multiple shapes including two or more shapes having centers that are less than a threshold distance from one another (¶¶ 0206-0208 [detecting…line…shape; with 0212-0214 [determine…distance…threshold value…does not exceed the threshold…midpoint], 0041 [recognize what are various hand drawn images resides in the software code as sets of rules and as real time algorithms; with 0022 [use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects], and 0214 [a vertex determination is made if the slice angle…exceeds the current threshold angle… past the region…which is at (or close to) the midpoint…places the vertex at the midpoint]]; Jaeger’s claims 56-58 [showing workflow – joining…in a utilitarian sequence…joining step includes the step of drawing at least one line extending to and intersecting…in said sequence]);
generating a merged shape by merging the two or more shapes having the centers that are less than the threshold distance from one another (¶¶ 0091 [combination of two or more…shapes; with 0247 [example – although a rectangle or folder or triangle…combined…identification as a circle], 0328 [operates to combine other graphic objects]; with 0214 [a vertex determination is made if the slice angle…exceeds the current threshold angle… past the region…which is at (or close to) the midpoint…places the vertex at the midpoint]]; Jaeger’s claims 56-58 [showing workflow – joining…in a utilitarian sequence…joining step includes the step of drawing at least one line extending to and intersecting…in said sequence]]);
determining a relationship between the merged shape and an additional shape of the multiple shapes based on a relative positioning of the line with respect to the merged shape and the additional shape (see citations above and also see ¶¶ 0019 [context…location, proximity], 0024[rectangle (shape)…location…can affect other objects…drawing connecting lines; see with 0054 and 0063-0064 [arrows/lines…can be hand drawn to designate links (relationships) between any graphic object or group of objects (shapes) and any other graphic object or group of objects; see with 0065-0067 [example given – arrow/line…nature of the link]]]; see also 0320-0323);
generating, for display in a user interface, a structured workflow diagram including compute nodes representing the merged shape and the additional shape that are connected by a generated line representing the relationship (¶¶ 0064 [the definition(s) of arrows' functions can be determined by the graphical context of the arrows: the way the arrows are drawn, where the arrows are drawn, the span of the graphical context of the arrows, where the arrows start and where they end up, and as mentioned above, when they were drawn or when they were started and then completed. It is the graphical context which the arrow encompasses together with the user defined nature of an arrow (herein termed "arrow logics") that can define that arrow's function (workflow is described here). When one draws an arrow between certain types of graphical objects (nodes), different operations may be commanded, depending upon the context(s) of the arrow; see with 0065-0068 [example of this workflow – object recognized by this system may have its own info window that can be used for the user-specification…definitions, assignment(s), operations, functions, actions…one can link these two knobs by drawing an arrow between them. Note that the direction of the arrow between the knobs is crucial in defining the nature of the link…operation of the arrow…arrows generally represent actions that are governed by logics and as such, once an arrow is drawn and recognized and its action carried out (workflow)], 0056-0058 [user interface… graphic on a screen], 0024 [display on screen/interface]]); and
generating a structured workflow representing the structured workflow diagram (¶¶ 0038 [hand drawn graphics…used to create…path…flow…order of events, types of devices, groupings of devices, relationships between devices; with 0055+ [hand drawn…becomes…flow diagram]]);
compiling the structured workflow file into executable code (see citations above for claim 1 and see ¶¶ 0002 [computer…programmed…software; with 0014 [functions to be carried out by the software (code)], 0041 [software code as sets of rules and as real time algorithms; with 0072 [configurable hardware, processes, actions, functions and the like, with software being implemented]], 0147 [operating system of the machine that this software is running on, i.e., a PC]); and
running the executable code to perform a workflow defined by the human- drawn workflow diagram (see citations above and see with, for example, ¶¶ 0072 [hand drawing of objects which can then be turned into virtual or actual…processes (workflow), actions (workflows), functions and the like, with software being implemented…processes, actions, functions, etc. being created, controlled and/or initiated; with 0038 [hand drawn graphics…used to create…path…flow…order of events, types of devices, groupings of devices, relationships between devices], 0014 [functions to be carried out by the software (code)]]).
Jaeger does not explicitly state non-transitory computer-readable storage media (although Jaeger does disclose computing system storing instruction/software and data/information).
Analogous art Dillner discloses non-transitory computer-readable storage media (¶¶ 0259 [computer-readable medium, such as an optical or magnetic disk or memory chip]).
Therefore, it would be obvious to one of ordinary skill in the art to include in the system/method of Jaeger non-transitory computer-readable storage media as taught by analogous art Dillner since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Dillner would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141).
Jaeger does not explicitly state file.
Analogous art Dillner discloses files and flow diagrams (¶¶ 0063 [hand drawn…results in flow diagram; with 0075 [transformed to a file that specifies…diagram; with 0209 [diagrams usually result in the construction of a directed graph that appropriately represents the flow of data in the diagram]], 0192 [representation…visual…environment…converted into a program understood by the visual programming environment (such as a text file specifying a diagram)]]).
Therefore, it would be obvious to one of ordinary skill in the art to include in the system/method of Jaeger file as taught by analogous art Dillner in order to efficiently provide essential instructions to the computing environment since doing so could be performed readily by any person of ordinary skill in the art, with neither undue experimentation, nor risk of unexpected results (KSR-G/TSM); and also since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Dillner would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141).
As per claim 21, Jaeger discloses the method of claim 1, further comprising allocating processing and memory resources to the workflow, and running the executable code using the processing and memory resources (see citations above for claim 1 and see ¶¶ 0002 [computer…programmed…software; with 0014 [functions to be carried out by the software (code)], 0041 [software code as sets of rules and as real time algorithms; with 0072 [configurable hardware, processes, actions, functions and the like, with software being implemented]], 0147 [operating system of the machine that this software is running on, i.e., a PC], 0072 [hand drawing of objects which can then be turned into virtual or actual…processes (workflow), actions (workflows), functions and the like, with software being implemented…processes, actions, functions, etc. being created, controlled and/or initiated; 0092 [ time stamp stored with it that can store the timings of the drawing of each point in each segment of the object. So, the ability to store], 0212 [software…system stores…stores…data], with 0038 [hand drawn graphics…used to create…path…flow…order of events, types of devices, groupings of devices, relationships between devices], 0014 [functions to be carried out by the software (code)]]).
Jaeger does not explicitly state memory (although it does discuss storing data).
Analogous art Dillner discloses memory (¶¶ 0072 [stored in computer memory in the form of an adjacency matrix for further processing; with 0081 [shows how the memory resource is used]], 0238-0259).
Therefore, it would be obvious to one of ordinary skill in the art to include in Jaeger memory as taught by analogous art Dillner in order to efficiently process information to the computing environment since doing so could be performed readily by any person of ordinary skill in the art, with neither undue experimentation, nor risk of unexpected results (KSR-G/TSM); and also since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Dillner would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141).
As per claim 22, Jaeger discloses the method of claim 1, wherein generating the structured workflow diagram includes generating graphical programming logic (software) in which the compute nodes correspond to particular operations and the generated lines are directed edges indicating directionality of the relationships between the compute nodes (¶¶ 0020-0024 [direction…arrow logics (has compute nodes)…line logics…programming connections, functions, links and behaviors and many types of associations (relationships); with 0030, 0038 [relationships], 0054 [The starting and ending points…located by two pixels…one for the beginning and one for the end], 0059 [operability of each hand drawn object and their relationship to each other], 0256-0259, 0275 [movement…direction…shape…edge], 0328-0330 [direction…closed chopped triangle would have a line connecting the open ends of the cut lines for this triangle. By connecting such lines, the resulting chopped triangle will become a closed object]], 0063 [logic of connections between individual elements and between pieces of equipment is user defined. Arrows can be hand drawn to designate links between any graphic object or group of objects and any other graphic object or group of objects. These links are user-definable and can represent actions, functions, definitions and the like], 0190-0200).
As per claim 23, Jaeger discloses the method of claim 1, wherein generating the structured workflow file includes converting the structured workflow diagram to a configuration format (see citations above for claims 1-4 and 21-22 and see ¶¶ 0002 [converts…to…machine-intelligible format], 0146-0149 [conversion…drawing…converted to…pixel locations], 0221, 0233 [ converted to a detailed graph], 0322 [software of the invention converts the on-screen arrangement to appropriate code to operate a programmable system that generates all the drawn functions, enables the drawn controls, establishes the drawn functional connections, and produces a desired result])
Jaeger does not explicitly state file.
Analogous art Dillner discloses files and flow diagrams (¶¶ 0063 [hand drawn…results in flow diagram; with 0075 [transformed to a file that specifies…diagram; with 0209 [diagrams usually result in the construction of a directed graph that appropriately represents the flow of data in the diagram]], 0192 [representation…visual…environment…converted into a program understood by the visual programming environment (such as a text file specifying a diagram)]]).
Therefore, it would be obvious to one of ordinary skill in the art to include in the system/method of Jaeger file as taught by analogous art Dillner in order to efficiently provide essential instructions to the computing environment since doing so could be performed readily by any person of ordinary skill in the art, with neither undue experimentation, nor risk of unexpected results (KSR-G/TSM); and also since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Dillner would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141).
Claims 3-4 are rejected under 35 U.S.C. 103 as being unpatentable over Jaeger et al., (US 2002/0141643) in view of Dillner (US 2004/0090439), further in view of Wekel et al., (US 2021/0063578)
As per claim 3, Jaeger discloses the method of claim 1, and detecting the shapes and the lines within images of human-drawn objects (see citations above for claim 1 and see ¶¶ 0206-0208 [detecting…line…shape; with 0041 [recognize what are various hand drawn images resides in the software code as sets of rules and as real time algorithms; with 0022 [use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects ], 0055 [hand drawn…flow diagram]]; Jaeger’s claims 56-58 [showing workflow – joining…in a utilitarian sequence…joining step includes the step of drawing at least one line extending to and intersecting…in said sequence]).
However, neither Jaeger nor Dillner disclose using a convolutional neural network having been trained.
Analogous art Wekel discloses a convolutional neural network having been trained (¶¶ 0048-0051 [image…training…convolutional neural network (CNN)]).
Therefore, it would be obvious to one of ordinary skill in the art to include in Jaeger in view of Dillner convolutional neural network having been trained as taught by analogous art Wekel in order to efficiently and optimal process images (efficiently extract spatial features) using AI/ML models since doing so could be performed readily by any person of ordinary skill in the art, with neither undue experimentation, nor risk of unexpected results (KSR-G/TSM); and also since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Wekel (using CNNs AI/ML models is well-known in imaging processing) would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141).
As per claim 4, Jaeger discloses the method of claim 1, wherein detecting the shapes and the lines includes detecting the shapes and the lines in the human-drawn workflow diagram using circles and lines detection/recognition software on devices (see citations for claim 1 above and see ¶¶ 0006 [recognize hand drawn objects…inputting data on…screen], 0041 [hand drawn images are recognized…input device]; claim 51 [input device to freehand draw…hand drawn entry into a functional device], 0206-0208 [detecting…line…shape; with 0041 [recognize what are various hand drawn images resides in the software code as sets of rules and as real time algorithms; with 0022 [use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects ], 0055 [hand drawn…flow diagram]]; Jaeger’s claims 56-58 [showing workflow – joining…in a utilitarian sequence…joining step includes the step of drawing at least one line extending to and intersecting…in said sequence]).
However, neither Jaeger nor Dillner state using Hough Transforms.
Analogous art Wekel discloses Hough Transforms (¶¶ 0043 [image…geometric…Hough transform algorithm…applied]).
Therefore, it would be obvious to one of ordinary skill in the art to include in Jaeger in view of Dillner Hough Transforms as taught by analogous art Wekel in order to efficiently and optimal process images (efficiently and accurately detect shapes in digital images) since doing so could be performed readily by any person of ordinary skill in the art, with neither undue experimentation, nor risk of unexpected results (KSR-G/TSM); and also since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Wekel (Hough transforms is well-known in imaging processing for detecting shapes) would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141).
Conclusion
The prior art made of record on the PTO-892 and not relied upon is considered pertinent to applicant's disclosure. For example, some of the pertinent prior art is as follows:
Murakami (US 2003/0123698): Provides an image processing apparatus and method, which embed a digital watermark having strong robustness against geometrical transformation/edit processes such as rotation, scaling, translation, and the like, and an image processing apparatus and method, which can accurately extract the embedded digital watermark even after the conversion/edit process such as rotation of an image or the like. A Fourier transformer (901) converts an image input from an image input unit (900) into amplitude component data, and phase component data. An envelope ring pattern generator (902) generates an embedding pattern of digital watermark information, which is expressed by a set of lines that define a circle as an envelope, on the amplitude component data. An envelope ring pattern embedding unit (903) embeds digital watermark information at predetermined positions on the line.
Wang et al., (US 2014/0328544): Discusses a sketch-based image recognition system that may generate a model for identifying a subject of a sketch. The model is formed from a plurality of images having visual features similar to the visual features of the sketch. The model may include object topics representative of categories which may correspond to the subject of the sketch and shape topics representative of the visual features of the sketch.
Schaefer et al., (US 2023/0062484): Provides for converting a diagram into a digital diagram format. A diagram is received as an image file and a plurality of recognition stages produce a final diagram based on predicted information from one or more of the recognition stages. The plurality of recognition stages including a shape detection stage for detecting a plurality of shapes and at least one arrow detection stage in which arrows are detected as relations between pairs of shapes. The final diagram is generated based on the predicted information and is converted into the digital diagram format compatible with a diagram modeling language.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GURKANWALJIT SINGH whose telephone number is (571)270-5392. The examiner can normally be reached on M-F 8:30-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Brian Epstein can be reached on 571-270-5389. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Gurkanwaljit Singh/
Primary Examiner, Art Unit 3625
Prosecution Insights