Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This non-final Office action is in response to applicant’s communication received on July 12, 2024, wherein claims 1-20 are currently pending.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Regarding Step 1 (MPEP 2106.03) of the subject matter eligibility test per MPEP 2106.03, Claims 1-12 are directed to a method (i.e., process), claims 13-16 are directed to a system (i.e. machine), and claims 17-20 are directed to non-transitory computer readable medium (i.e. product or article of manufacture). Accordingly, all claims are directed to one of the four statutory categories of invention.
(Under Step 2) The claimed invention is directed to an abstract idea without significantly more.
(Under Step 2A, Prong 1 (MPEP 2106.04)) The independent claims (1, 13, 17) recite receiving/obtaining information/data (where the information itself is abstract in nature – e.g. diagram, workflow, shapes, lines, placement, etc.,), data/information analysis/manipulation (comparing information (e.g. regarding shapes, lines, workflows, etc.,), organizing data/information and determining relationships (between data/information), etc.,) to determine more data/information, possibly obtaining more abstract information/data, and providing this determined data/information for further analysis and reproduction/displaying. The limitations of the independent claims (1, 13, 17), under the broadest reasonable interpretation, covers methods of organizing human activity (managing workflows (interaction and relationships between elements of the workflow)) and mental process (observing (diagram and drawings with shapes and lines, etc.,) and evaluating the information regarding human drawn information/diagrams and creating/recreating workflows using generic computing elements using received information). If a claims limitation, under its broadest reasonable interpretation, covers the performance of the limitation as fundamental economic principles or practices (including hedging, insurance, mitigating risk); commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations); managing personal behavior or relationships or interactions between people (including scheduling, social activities, teaching, and following rules or instructions), then it falls within the “organizing human activities” grouping of abstract ideas. (MPEP 2106.04). If claim limitations, under its broadest reasonable interpretation, cover the performance of the limitation as concepts performed in the human mind (including an observation, evaluation, judgment, opinion), the claim limitations fall within the Mental process grouping of abstract ideas. (MPEP 2106.04).
Accordingly, since Applicant's claims fall under organizing human activities grouping and mental process grouping, the claims recite an abstract idea.
(Under Step 2A, prong 2 (MPEP 2106.04(d))) This judicial exception is not integrated into a practical application because but for the recitation of well-known generic/general-purpose computing/technology components/elements/terms (computing device, display in user interface, file (in independent claim 1); system, processor, memory, display in user interface, file (in independent claim 13); non-transitory computer-readable storage media, processing device, display in user interface, file (in independent claim 17)), in the context of the independent claims (1, 13, 17), the claims encompass the above stated abstract idea (organizing human activity (managing workflows (interaction and relationships between elements of the workflow)) and mental process (observing (diagram and drawings with shapes and lines, etc.,) and evaluating the information regarding human drawn information/diagrams and creating/recreating workflows using generic computing elements using received information)).
As shown above, the independent claims (1, 13, 17) recite generic/general-purpose computing/technology components/elements/terms/limitations (computing device, display in user interface, file (in independent claim 1); system, processor, memory, display in user interface, file (in independent claim 13); non-transitory computer-readable storage media, processing device, display in user interface, file (in independent claim 17)) which are recited at a high level of generality performing generic/general purpose computer/computing functions. (MPEP 2106.04). The generic/general-purpose computing/technology components/elements/terms/limitations are no more than mere instructions to apply the judicial exception (the above abstract idea – organizing human activity (managing workflows (interaction and relationships between elements of the workflow)) and mental process (observing (diagram and drawings with shapes and lines, etc.,) and evaluating the information regarding human drawn information/diagrams and creating/recreating workflows using generic computing elements using received information)) in an apply-it fashion using generic/general-purpose computing/technology components/elements/terms/limitations (computing device, display in user interface, file (in independent claim 1); system, processor, memory, display in user interface, file (in independent claim 13); non-transitory computer-readable storage media, processing device, display in user interface, file (in independent claim 17)). The CAFC has stated that it is not enough, however, to merely improve abstract processes by invoking a computer merely as a tool. Customedia Techs., LLC v. Dish Network Corp., 951 F.3d 1359, 1364 (Fed. Cir. 2020). The focus of the claims is simply to use computers and a familiar network as a tool to perform abstract processes (discussed above) involving simple information exchange. Carrying out abstract processes involving information exchange is an abstract idea. See, e.g., BSG, 899 F.3d at 1286; SAP America, 898 F.3d at 1167-68; Affinity Labs of Tex., LLC v. DIRECTV, LLC, 838 F.3d 1253, 1261-62 (Fed. Cir. 2016). And use of standard computers and networks to carry out those functions—more speedily, more efficiently, more reliably—does not make the claims any less directed to that abstract idea. See Alice Corp., 573 U.S. at 222-25; Customedia, 951 F.3d at 1364; Trading Techs. Int'l, Inc. v. IBG LLC, 921 F.3d 1084, 1092-93 (Fed. Cir. 2019); SAP America, 898 F.3d at 1167; Intellectual Ventures I LLC v. Symantec Corp., 838 F.3d 1307, 1314 (Fed. Cir. 2016); Electric Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1353, 1355 (Fed. Cir. 2016); Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 1370 (Fed. Cir. 2015); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355 (Fed. Cir. 2014). Accordingly, the additional elements (computing device, display in user interface, file (in independent claim 1); system, processor, memory, display in user interface, file (in independent claim 13); non-transitory computer-readable storage media, processing device, display in user interface, file (in independent claim 17)) do not integrate the abstract idea in to a practical application because it does not impose any meaningful limits on practicing the abstract idea – i.e. they are just post-solution/extra-solution activities.
(Under Step 2B (MPEP 2106.05)) The independent claims (1, 13, 17) do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the claims do not recite an improvement to another technology or technical field, an improvement to the functioning of the computer itself, or meaningful limitations beyond generally linking the use of an abstract idea to a particular technological environment. The independent claims recite using known generic/general-purpose computing/technology components/elements/terms/limitations (computing device, display in user interface, file (in independent claim 1); system, processor, memory, display in user interface, file (in independent claim 13); non-transitory computer-readable storage media, processing device, display in user interface, file (in independent claim 17)). For the role of a computer in a computer implemented invention to be deemed meaningful in the context of this analysis, it must involve more than performance of "well-understood, routine, [and] conventional activities previously known to the industry." Alice Corp. v. CLS Bank Int'l, 110 USPQ2d 1976 (U.S. 2014), at 2359 (quoting Mayo, 132 S. Ct. at 1294 (internal quotation marks and brackets omitted)). These activities as claimed by the Applicant are all well-known and routine tasks in the field of art – as can been seen in the specification of Applicant’s application (for example, see Applicant’s specification at, for example, figure 4 and Pages 24-25 [where Applicant recites general-purpose/generic computers/processors/etc., and generic/general-purpose computing components/devices/etc., in Applicant’s specification]) and/or the specification of the below cited art (used in the rejection below and on the PTO-892) and/or also as noted in the court cases in §2106.05 in the MPEP. Further, "the mere recitation of a generic computer cannot transform a patent ineligible abstract idea into a patent-eligible invention." Alice at 2358. None of the hardware offers a meaningful limitation beyond generally linking the system to a particular technological environment, that is, implementation via computers. Adding generic computer components to perform generic functions that are well‐understood, routine and conventional, such as gathering data, performing calculations, and outputting a result would not transform the claims into eligible subject matter. Abstract ideas are excluded from patent eligibility based on a concern that monopolization of the basic tools of scientific and technological work might impede innovation more than it would promote it. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the claims require no more than a generic computer to perform generic computer functions. The additional elements (computing device, display in user interface, file (in independent claim 1); system, processor, memory, display in user interface, file (in independent claim 13); non-transitory computer-readable storage media, processing device, display in user interface, file (in independent claim 17)) or combination of elements in the claims other than the abstract idea per se amounts to no more than: (i) mere instructions to implement the idea on a computer, and/or (ii) recitation of generic computer structure that serves to perform generic computer functions that are well-understood, routine, and conventional activities previously known to the pertinent industry. Applicant is directed to the following citations and references: Digitech Image., LLC v. Electronics for Imaging, Inc. (758 F.3d 1344 (2014) discussing U.S. Patent No. 6,128,415); and (2) Federal register/Vol. 79, No 241 issued on December 16, 2014, page 74629, column 2, Gottschalk v. Benson. Viewed as a whole, the independent claims do not purport to improve the functioning of the computer itself, or to improve any other technology or technical field. Use of an unspecified, generic computer does not transform an abstract idea into a patent-eligible invention. Thus, the independent claims do not amount to significantly more than the abstract idea itself. See Alice Corp. v. CLS Bank Int'l, 110 USPQ2d 1976 (U.S. 2014).
The dependent claims (2-12, 14-16, 18-20) further define the independent claims and merely narrow the described abstract idea, but not adding significantly more than the abstract idea. The dependent claims either individually or in combination are merely an extension of the abstract idea itself. The above rejection discussed for the independent claims fully applies to the dependent claims.
The dependent claims (2-12, 14-16, 18-20) further state using obtained data/information (where the information itself is abstract in nature – e.g. diagram, workflow, shapes, lines, placement, etc.,), data/information analysis/manipulation (comparing information (e.g. regarding shapes, lines, workflows, etc.,), organizing data/information and determining relationships (between data/information), etc.,) to determine more data/information, possibly obtaining more abstract information/data, and providing this determined data/information for further analysis and reproduction/displaying. These dependent claims also cover methods of organizing human activity (managing workflows (interaction and relationships between elements of the workflow)) and mental process (observing (diagram and drawings with shapes and lines, etc.,) and evaluating the information regarding human drawn information/diagrams and creating/recreating workflows using generic computing elements using received information).
This judicial exception is not integrated into a practical application because the claims and specification recite additional elements as generic/general-purpose computing/technology components/elements/terms/limitations (convolutional neural network (term only stated but no technical details provided on how it functions as per Applicant’s claims), trained/training (only stated but no technical details provided on how it functions as per Applicant’s claims), optical character recognition, file, executable code (software), (in claim 1’s dependent claims 2-12); system (in claim 13’s dependent claims 14-16); non-transitory computer-readable storage media, file, executable code (software) (in claim 17’s dependent claims 18-20)) performing generic computer/computing/technology functions. (MPEP 2106.04). The dependent claims merely use the same general technological environment and instructions as the independent claims above to implement the abstract idea. The generic/general-purpose computing/technology components/elements/terms/limitations are no more than mere instructions to apply the judicial exception (the above abstract idea – organizing human activity (managing workflows (interaction and relationships between elements of the workflow)) and mental process (observing (diagram and drawings with shapes and lines, etc.,) and evaluating the information regarding human drawn information/diagrams and creating/recreating workflows using generic computing elements using received information)) in an apply-it fashion using generic/general-purpose computing/technology components/elements/terms/limitations (convolutional neural network (term only stated but no technical details provided on how it functions as per Applicant’s claims), trained/training (only stated but no technical details provided on how it functions as per Applicant’s claims), optical character recognition, file, executable code (software), (in claim 1’s dependent claims 2-12); system (in claim 13’s dependent claims 14-16); non-transitory computer-readable storage media, file, executable code (software) (in claim 17’s dependent claims 18-20)). Hence, the additional elements (convolutional neural network (term only stated but no technical details provided on how it functions as per Applicant’s claims), trained/training (only stated but no technical details provided on how it functions as per Applicant’s claims), optical character recognition, file, executable code (software), (in claim 1’s dependent claims 2-12); system (in claim 13’s dependent claims 14-16); non-transitory computer-readable storage media, file, executable code (software) (in claim 17’s dependent claims 18-20)) do not integrate the abstract idea in to a practical application because they does not impose any meaningful limits on practicing the abstract idea – i.e. they are just post-solution/extra-solution activities.
Also, the dependent claims (2-12, 14-16, 18-20) either individually or in combination are merely an extension of the abstract idea itself and the dependent claims (similar to the independent claims) do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the claims require no more than a generic computer to perform generic computer functions. The additional elements (convolutional neural network (term only stated but no technical details provided on how it functions as per Applicant’s claims), trained/training (only stated but no technical details provided on how it functions as per Applicant’s claims), optical character recognition, file, executable code (software), (in claim 1’s dependent claims 2-12); system (in claim 13’s dependent claims 14-16); non-transitory computer-readable storage media, file, executable code (software) (in claim 17’s dependent claims 18-20)) or combination of elements in the dependent claims other than the abstract idea per se amounts to no more than: (i) mere instructions to implement the idea on a computer, and/or (ii) recitation of generic computer structure that serves to perform generic computer functions that are well-understood, routine, and conventional activities previously known to the pertinent industry. Applicant is directed to the following citations and references: Digitech Image., LLC v. Electronics for Imaging, Inc. (758 F.3d 1344 (2014) discussing U.S. Patent No. 6,128,415); and (2) Federal register/Vol. 79, No 241 issued on December 16, 2014, page 74629, column 2, Gottschalk v. Benson. Viewed as a whole, dependent claims do not purport to improve the functioning of the computer itself, or to improve any other technology or technical field. Use of an unspecified, generic computer does not transform an abstract idea into a patent-eligible invention. Thus, the dependent claims do not amount to significantly more than the abstract idea itself. See Alice Corp. v. CLS Bank Int'l, 110 USPQ2d 1976 (U.S. 2014).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2 and 5-20 are rejected under 35 U.S.C. 103 as being unpatentable over Jaeger et al., (US 2002/0141643) in view of Dillner (US 2004/0090439).
As per claim 1, Jaeger discloses a method implemented by at least one computing device, the method comprising:
receiving an image of a human-drawn workflow diagram (¶¶ 0006 [recognize hand drawn objects…inputting data on…screen], 0041 [hand drawn images are recognized…input device]; claim 51 [input device to freehand draw…hand drawn entry into a functional device]);
detecting, using one or more image recognition algorithms, shapes and lines in the human-drawn workflow diagram (¶¶ 0206-0208 [detecting…line…shape; with 0041 [recognize what are various hand drawn images resides in the software code as sets of rules and as real time algorithms; with 0022 [use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects ], 0055 [hand drawn…flow diagram]]; Jaeger’s claims 56-58 [showing workflow – joining…in a utilitarian sequence…joining step includes the step of drawing at least one line extending to and intersecting…in said sequence]);
extending the lines detected in the human-drawn workflow diagram (Jaeger’s claim 58 [joining step includes the step of drawing at least one line extending], claim 79 [line that extends]; also see ¶¶ 0403);
determining relationships between the shapes based on relative positionings of the extended lines with respect to the shapes (see citations above and also see ¶¶ 0019 [context…location, proximity], 0024[rectangle (shape)…location…can affect other objects…drawing connecting lines; see with 0054 and 0063-0064 [arrows/lines…can be hand drawn to designate links (relationships) between any graphic object or group of objects (shapes) and any other graphic object or group of objects; see with 0065-0067 [example given – arrow/line…nature of the link]]]; see also 0320-0323);
generating, for display in a user interface, a structured workflow diagram including compute nodes representing the shapes that are connected by generated lines representing the relationships (¶¶ 0064 [the definition(s) of arrows' functions can be determined by the graphical context of the arrows: the way the arrows are drawn, where the arrows are drawn, the span of the graphical context of the arrows, where the arrows start and where they end up, and as mentioned above, when they were drawn or when they were started and then completed. It is the graphical context which the arrow encompasses together with the user defined nature of an arrow (herein termed "arrow logics") that can define that arrow's function (workflow is described here). When one draws an arrow between certain types of graphical objects (nodes), different operations may be commanded, depending upon the context(s) of the arrow; see with 0065-0068 [example of this workflow – object recognized by this system may have its own info window that can be used for the user-specification…definitions, assignment(s), operations, functions, actions…one can link these two knobs by drawing an arrow between them. Note that the direction of the arrow between the knobs is crucial in defining the nature of the link…operation of the arrow…arrows generally represent actions that are governed by logics and as such, once an arrow is drawn and recognized and its action carried out (workflow)], 0056-0058 [user interface… graphic on a screen], 0024 [display on screen/interface]]); and
generating a structured workflow representing the structured workflow diagram (¶¶ 0038 [hand drawn graphics…used to create…path…flow…order of events, types of devices, groupings of devices, relationships between devices; with 0055+ [hand drawn…becomes…flow diagram]]).
Jaeger does not explicitly state file.
Analogous art Dillner discloses files and flow diagrams (¶¶ 0063 [hand drawn…results in flow diagram; with 0075 [transformed to a file that specifies…diagram; with 0209 [diagrams usually result in the construction of a directed graph that appropriately represents the flow of data in the diagram]], 0192 [representation…visual…environment…converted into a program understood by the visual programming environment (such as a text file specifying a diagram)]]).
Therefore, it would be obvious to one of ordinary skill in the art to include in the system/method of Jaeger file as taught by analogous art Dillner in order to efficiently provide essential instructions to the computing environment since doing so could be performed readily by any person of ordinary skill in the art, with neither undue experimentation, nor risk of unexpected results (KSR-G/TSM); and also since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Dillner would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141; and also see (1) 2007 Examination Guidelines for Determining Obviousness Under 35 U.S.C. 103 in View of the Supreme Court Decision in KSR International Co. v. Teleflex Inc. - Federal Register, Vol. 72, No. 195, October 10, 2007, pages 57526-57535; (2) 2010 Examination Guidelines Updated Developments in the Obviousness Inquiry After KSR v. Teleflex. -Federal Register, Vol. 75, No. 169, September 01, 2010, pages 53643-53660; and (3) materials posted at https://www.uspto.gov/patent/laws-and-regulations/examination-policy/examination-guidelines-training-materials-view-ksr).
As per claim 2, Jaeger discloses the method of claim 1, wherein detecting the shapes and the lines includes extracting coordinates of the shapes and the lines within the image, and determining the relationships is based on the coordinates ¶¶ 0006 [location of drawing], 0019 [context…location, proximity], 0024[rectangle (shape)…location…can affect other objects…drawing connecting lines; see with 0053-0054 [location…image…exact location of…image] and 0063-0064 [arrows/lines…can be hand drawn to designate links (relationships) between any graphic object or group of objects (shapes) and any other graphic object or group of objects; see with 0065-0067 [example given – arrow/line…nature of the link]]]; see also 0320-0323).
As per claim 5, Jaeger discloses the method of claim 1, wherein detecting the lines includes detecting multiple lines having midpoints that are less than a threshold distance from one another (¶¶ 0206-0208 [detecting…line…shape; with 0041 [recognize what are various hand drawn images resides in the software code as sets of rules and as real time algorithms; with 0022 [use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects ], 0055 [hand drawn…flow diagram], and 0214 [a vertex determination is made if the slice angle…exceeds the current threshold angle… past the region…which is at (or close to) the midpoint…places the vertex at the midpoint]]; Jaeger’s claims 56-58 [showing workflow – joining…in a utilitarian sequence…joining step includes the step of drawing at least one line extending to and intersecting…in said sequence]) and
retaining a single line of the multiple lines, wherein determining the relationships includes determining a relationship between two shapes based on the single line (¶¶ 0064 [the definition(s) of arrows' functions can be determined by the graphical context of the arrows: the way the arrows are drawn, where the arrows are drawn, the span of the graphical context of the arrows, where the arrows start and where they end up, and as mentioned above, when they were drawn or when they were started and then completed. It is the graphical context which the arrow encompasses together with the user defined nature of an arrow (herein termed "arrow logics") that can define that arrow's function (workflow is described here). When one draws an arrow between certain types of graphical objects (nodes), different operations may be commanded, depending upon the context(s) of the arrow; see with 0139 [enabling a single…graphic/line to represent and implement], 0065-0068 [example of this workflow – object recognized by this system may have its own info window that can be used for the user-specification…definitions, assignment(s), operations, functions, actions…one can link these two knobs by drawing an arrow between them…the direction of the arrow between the knobs is crucial in defining the nature of the link…operation of the arrow…arrows generally represent actions that are governed by logics and as such, once an arrow is drawn and recognized and its action carried out (workflow)]]).
As per claim 18, claim 18 discloses substantially similar limitations as claim 5 above; and therefore claim 18 is rejected under the same rationale and reasoning as presented above for claim 5.
As per claim 6, Jaeger discloses the method of claim 1, further comprising removing at least one shape detected in the human-drawn workflow diagram based on a degree of difference between a size of the at least one shape relative to an average size of the shapes in the human-drawn workflow diagram, wherein determining the relationships is based on the relative positionings of the lines with respect to the shapes excluding the at least one shape (¶¶ 0063 [lines/shapes…not required; with 0330 [editing…chop tool…cuts…line/lines], 0155 [objects…eliminated; with 0210 [not allowed…size…eliminated], 0249 [size limits…eliminate…object…size differential may be employed to distinguish the two objects]; 0245-0247 [elimination of…shapes], 0139 [enabling a single…graphic/line to represent and implement], 0024 [rectangle (shape)…location…can affect other objects…drawing connecting lines; see with 0054 and 0063-0064 [arrows/lines…can be hand drawn to designate links (relationships) between any graphic object or group of objects (shapes) and any other graphic object or group of objects; see with 0065-0067 [example given – arrow/line…nature of the link]]]; see also 0320-0323).
As per claim 14, claim 14 discloses substantially similar limitations as claim 6 above; and therefore claim 14 is rejected under the same rationale and reasoning as presented above for claim 6.
As per claim 7, Jaeger discloses the method of claim 1, wherein detecting the shapes includes detecting multiple shapes having centers that are within a threshold distance from one another and generating a merged shape by merging the multiple shapes, wherein determining the relationships includes determining a relationship between the merged shape and an additional shape (¶¶ 0206-0208 [detecting…line…shape; with 0212-0214 [determine…distance…threshold value…does not exceed the threshold…midpoint], 0041 [recognize what are various hand drawn images resides in the software code as sets of rules and as real time algorithms; with 0022 [use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects], and 0214 [a vertex determination is made if the slice angle…exceeds the current threshold angle… past the region…which is at (or close to) the midpoint…places the vertex at the midpoint]]; Jaeger’s claims 56-58 [showing workflow – joining…in a utilitarian sequence…joining step includes the step of drawing at least one line extending to and intersecting…in said sequence], se with 0091 [combination of two or more…shapes; with 0247 [example – although a rectangle or folder or triangle…combined…identification as a circle], 0328 [operates to combine other graphic objects]], 0019 [context…location, proximity], 0024 [rectangle (shape)…location…can affect other objects…drawing connecting lines; see with 0054 and 0063-0064 [arrows/lines…can be hand drawn to designate links (relationships) between any graphic object or group of objects (shapes) and any other graphic object or group of objects; see with 0065-0067 [example given – arrow/line…nature of the link]]]; see also 0320-0323).
As per claim 8, Jaeger discloses the method of claim 1, wherein detecting the lines includes detecting arrows and directions in which the arrows are pointed, wherein determining the relationships is further based on the directions (see citations above for claim 1 and see ¶¶ 0022 [use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects; with 0054 [arrow…starting and ending points], 0062-0066 [arrow logics…direction of the arrow…arrowhead is pointing]]).
As per claim 9, Jaeger discloses the method of claim 1, further comprising: detecting, using an algorithm, characters in the human-drawn workflow diagram; assigning the characters to the shapes and the lines based on relative positionings of the characters with respect to the shapes and the lines, respectively (see citations above for claim 1 and see ¶¶ 0206-0208 [detecting…line…shape; with 0041 [recognize what are various hand drawn images resides in the software code as sets of rules and as real time algorithms; with 0022-0025 [use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects…assign colors…color defines function and/or assignment and/or properties and/or actions], 0055 [hand drawn…flow diagram]], 0064 [the definitions of arrows' functions can be determined by the graphical context of the arrows: the way the arrows are drawn, where the arrows are drawn, the span of the graphical context of the arrows, where the arrows start and where they end up, and as mentioned above, when they were drawn or when they were started and then completed. It is the graphical context which the arrow encompasses together with the user defined nature of an arrow (herein termed "arrow logics") that can define that arrow's function (workflow is described here). When one draws an arrow between certain types of graphical objects, different operations may be commanded, depending upon the context(s) of the arrow; see with 0065-0068 [example of this workflow – object recognized by this system may have its own info window that can be used for the user-specification…definitions, assignments, operations, functions, actions…one can link these two knobs by drawing an arrow between them. Note that the direction of the arrow between the knobs is crucial in defining the nature of the link…operation of the arrow…arrows generally represent actions that are governed by logics and as such, once an arrow is drawn and recognized and its action carried out], 0056-0058 [user interface… graphic on a screen], 0024 [display on screen/interface]]);
associating the compute nodes with operation types corresponding to the characters assigned to the shapes; and associating the relationships with relationship types corresponding to the characters assigned to the lines (see citations above for claim 1 and see ¶¶ 0064 [the definitions of arrows' functions can be determined by the graphical context of the arrows: the way the arrows are drawn, where the arrows are drawn, the span of the graphical context of the arrows, where the arrows start and where they end up, and as mentioned above, when they were drawn or when they were started and then completed. It is the graphical context which the arrow encompasses together with the user defined nature of an arrow (herein termed "arrow logics") that can define that arrow's function (workflow is described here). When one draws an arrow between certain types of graphical objects (nodes), different operations may be commanded, depending upon the context(s) of the arrow; see with 0065-0068 [example of this workflow – object recognized by this system may have its own info window that can be used for the user-specification…definitions, assignments, operations, functions, actions…one can link these two knobs by drawing an arrow between them. Note that the direction of the arrow between the knobs is crucial in defining the nature of the link…operation of the arrow…arrows generally represent actions that are governed by logics and as such, once an arrow is drawn and recognized and its action carried out (workflow)]).
Jaeger does not state optical character recognition.
Analogous art Dillner discloses optical character recognition (¶¶ 0261-0262 [optical character recognition…OCR]).
Therefore, it would be obvious to one of ordinary skill in the art to include in the system/method of Jaeger optical character recognition as taught by analogous art Dillner in order to efficiently process images (machine vision tasks) since doing so could be performed readily by any person of ordinary skill in the art, with neither undue experimentation, nor risk of unexpected results (KSR-G/TSM); and also since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Dillner (optical character recognition (OCR) is old and well-known in image processing) would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141; and also see (1) 2007 Examination Guidelines for Determining Obviousness Under 35 U.S.C. 103 in View of the Supreme Court Decision in KSR International Co. v. Teleflex Inc. - Federal Register, Vol. 72, No. 195, October 10, 2007, pages 57526-57535; (2) 2010 Examination Guidelines Updated Developments in the Obviousness Inquiry After KSR v. Teleflex. -Federal Register, Vol. 75, No. 169, September 01, 2010, pages 53643-53660; and (3) materials posted at https://www.uspto.gov/patent/laws-and-regulations/examination-policy/examination-guidelines-training-materials-view-ksr).
As per claim 10, Jaeger discloses the method of claim 1, further comprising: detecting, using an algorithm, characters in the human-drawn workflow diagram; assigning the characters to the shapes and the lines based on relative positionings of the characters with respect to the shapes and the lines, respectively (see citations above for claim 1 and see ¶¶ 0206-0208 [detecting…line…shape; with 0041 [recognize what are various hand drawn images resides in the software code as sets of rules and as real time algorithms; with 0022-0025 [use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects…assign colors…color defines function and/or assignment and/or properties and/or actions], 0055 [hand drawn…flow diagram]], 0064 [the definitions of arrows' functions can be determined by the graphical context of the arrows: the way the arrows are drawn, where the arrows are drawn, the span of the graphical context of the arrows, where the arrows start and where they end up, and as mentioned above, when they were drawn or when they were started and then completed. It is the graphical context which the arrow encompasses together with the user defined nature of an arrow (herein termed "arrow logics") that can define that arrow's function (workflow is described here). When one draws an arrow between certain types of graphical objects, different operations may be commanded, depending upon the context(s) of the arrow; see with 0065-0068 [example of this workflow – object recognized by this system may have its own info window that can be used for the user-specification…definitions, assignments, operations, functions, actions…one can link these two knobs by drawing an arrow between them. Note that the direction of the arrow between the knobs is crucial in defining the nature of the link…operation of the arrow…arrows generally represent actions that are governed by logics and as such, once an arrow is drawn and recognized and its action carried out], 0056-0058 [user interface… graphic on a screen], 0024 [display on screen/interface]]); and
labeling the compute nodes and the generated lines with identifiers in the structured workflow diagram based on the characters assigned to the shapes and the lines, respectively (see citations above for claim 1 and see ¶¶ 0064 [the definitions of arrows' functions can be determined by the graphical context of the arrows: the way the arrows are drawn, where the arrows are drawn, the span of the graphical context of the arrows, where the arrows start and where they end up, and as mentioned above, when they were drawn or when they were started and then completed. It is the graphical context which the arrow encompasses together with the user defined nature of an arrow (herein termed "arrow logics") that can define that arrow's function (workflow is described here). When one draws an arrow between certain types of graphical objects (nodes), different operations may be commanded, depending upon the context(s) of the arrow; see with 0065-0068 [example of this workflow – object recognized by this system may have its own info window that can be used for the user-specification…definitions, assignments, operations, functions, actions…one can link these two knobs by drawing an arrow between them. Note that the direction of the arrow between the knobs is crucial in defining the nature of the link…operation of the arrow…arrows generally represent actions that are governed by logics and as such, once an arrow is drawn and recognized and its action carried out (workflow)]).
Jaeger does not state optical character recognition.
Analogous art Dillner discloses optical character recognition (¶¶ 0261-0262 [optical character recognition…OCR]).
Therefore, it would be obvious to one of ordinary skill in the art to include in the system/method of Jaeger optical character recognition as taught by analogous art Dillner in order to efficiently process images (machine vision tasks) since doing so could be performed readily by any person of ordinary skill in the art, with neither undue experimentation, nor risk of unexpected results (KSR-G/TSM); and also since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Dillner (optical character recognition (OCR) is old and well-known in image processing) would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141; and also see (1) 2007 Examination Guidelines for Determining Obviousness Under 35 U.S.C. 103 in View of the Supreme Court Decision in KSR International Co. v. Teleflex Inc. - Federal Register, Vol. 72, No. 195, October 10, 2007, pages 57526-57535; (2) 2010 Examination Guidelines Updated Developments in the Obviousness Inquiry After KSR v. Teleflex. -Federal Register, Vol. 75, No. 169, September 01, 2010, pages 53643-53660; and (3) materials posted at https://www.uspto.gov/patent/laws-and-regulations/examination-policy/examination-guidelines-training-materials-view-ksr).
As per claim 11, Jaeger discloses the method of claim 1, wherein detecting the shapes includes detecting different shape types in the human-drawn workflow diagram (see citations for claim 1 above and see, for example, ¶¶ 0089 [rectangles, squares, equilateral triangles, etc.,], 0200 [circle or a rectangle or triangle], 0218 [hand drawn…type of…shape…rectangle…triangle…folder]),
wherein generating the structured workflow includes associating the compute nodes in the structured workflow with different operation types corresponding to the different shape types (see citations for claim 1 above and see, for example, ¶¶ 0064 [the definitions of arrows' functions can be determined by the graphical context of the arrows: the way the arrows are drawn, where the arrows are drawn, the span of the graphical context of the arrows, where the arrows start and where they end up, and as mentioned above, when they were drawn or when they were started and then completed. It is the graphical context which the arrow encompasses together with the user defined nature of an arrow (herein termed "arrow logics") that can define that arrow's function (workflow is described here). When one draws an arrow between certain types of graphical objects (nodes), different operations may be commanded, depending upon the context(s) of the arrow; see with 0065-0068 [example of this workflow – object recognized by this system may have its own info window that can be used for the user-specification…definitions, assignments, operations, functions, actions…one can link these two knobs by drawing an arrow between them. Note that the direction of the arrow between the knobs is crucial in defining the nature of the link…operation of the arrow…arrows generally represent actions that are governed by logics and as such, once an arrow is drawn and recognized and its action carried out (workflow)]).
Jaeger does not explicitly state file.
Analogous art Dillner discloses files and flow diagrams (¶¶ 0063 [hand drawn…results in flow diagram; with 0075 [transformed to a file that specifies…diagram; with 0209 [diagrams usually result in the construction of a directed graph that appropriately represents the flow of data in the diagram]], 0192 [representation…visual…environment…converted into a program understood by the visual programming environment (such as a text file specifying a diagram)]]).
Therefore, it would be obvious to one of ordinary skill in the art to include in the system/method of Jaeger file as taught by analogous art Dillner in order to efficiently provide essential instructions to the computing environment since doing so could be performed readily by any person of ordinary skill in the art, with neither undue experimentation, nor risk of unexpected results (KSR-G/TSM); and also since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Dillner would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141; and also see (1) 2007 Examination Guidelines for Determining Obviousness Under 35 U.S.C. 103 in View of the Supreme Court Decision in KSR International Co. v. Teleflex Inc. - Federal Register, Vol. 72, No. 195, October 10, 2007, pages 57526-57535; (2) 2010 Examination Guidelines Updated Developments in the Obviousness Inquiry After KSR v. Teleflex. -Federal Register, Vol. 75, No. 169, September 01, 2010, pages 53643-53660; and (3) materials posted at https://www.uspto.gov/patent/laws-and-regulations/examination-policy/examination-guidelines-training-materials-view-ksr).
As per claim 12, Jaeger discloses the method of claim 1, further comprising compiling the structured workflow into executable code, and running the executable code (see citations above for claim 1 and see ¶¶ 0002 [computer…programmed…software; with 0014 [functions to be carried out by the software (code)], 0041 [software code as sets of rules and as real time algorithms; with 0072 [configurable hardware, processes, actions, functions and the like, with software being implemented]], 0147 [operating system of the machine that this software is running on, i.e., a PC]).
Jaeger does not explicitly state file.
Analogous art Dillner discloses files and flow diagrams (¶¶ 0063 [hand drawn…results in flow diagram; with 0075 [transformed to a file that specifies…diagram; with 0209 [diagrams usually result in the construction of a directed graph that appropriately represents the flow of data in the diagram]], 0192 [representation…visual…environment…converted into a program understood by the visual programming environment (such as a text file specifying a diagram)]]).
Therefore, it would be obvious to one of ordinary skill in the art to include in the system/method of Jaeger file as taught by analogous art Dillner in order to efficiently provide essential instructions to the computing environment since doing so could be performed readily by any person of ordinary skill in the art, with neither undue experimentation, nor risk of unexpected results (KSR-G/TSM); and also since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Dillner would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141; and also see (1) 2007 Examination Guidelines for Determining Obviousness Under 35 U.S.C. 103 in View of the Supreme Court Decision in KSR International Co. v. Teleflex Inc. - Federal Register, Vol. 72, No. 195, October 10, 2007, pages 57526-57535; (2) 2010 Examination Guidelines Updated Developments in the Obviousness Inquiry After KSR v. Teleflex. -Federal Register, Vol. 75, No. 169, September 01, 2010, pages 53643-53660; and (3) materials posted at https://www.uspto.gov/patent/laws-and-regulations/examination-policy/examination-guidelines-training-materials-view-ksr).
As per claim 20, claim 20 discloses substantially similar limitations as claim 12 above; and therefore claim 20 is rejected under the same rationale and reasoning as presented above for claim 12.
As per claim 13, Jaeger discloses system, comprising: at least one processor; and storing instructions, which when executed by the at least one processor, cause the at least one processor to perform operations (¶¶ 0004 [system], 0028 [processor…hardware…software], 0093 [system…processor], 0184, 0212-0213 [system stores data], 0230-0235, 0325) including:
receiving an image of a human-drawn workflow diagram (¶¶ 0006 [recognize hand drawn objects…inputting data on…screen], 0041 [hand drawn images are recognized…input device]; claim 51 [input device to freehand draw…hand drawn entry into a functional device]);
detecting, using one or more image recognition algorithms, multiple shapes and multiple lines having midpoints that are less than a threshold distance from one another in the human-drawn workflow diagram (¶¶ 0206-0208 [detecting…line…shape; with 0041 [recognize what are various hand drawn images resides in the software code as sets of rules and as real time algorithms; with 0022 [use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects ], 0055 [hand drawn…flow diagram], and 0214 [a vertex determination is made if the slice angle…exceeds the current threshold angle… past the region…which is at (or close to) the midpoint…places the vertex at the midpoint]]; Jaeger’s claims 56-58 [showing workflow – joining…in a utilitarian sequence…joining step includes the step of drawing at least one line extending to and intersecting…in said sequence]);
retaining a single line of the multiple lines by removing one or more lines of the multiple lines (¶¶ 0063 [lines…not required; with 0330 [editing…chop tool…cuts…line/lines], 0139 [enabling a single…graphic/line to represent and implement]; and also 0206-0208 [measure the trend of a single segment…if one drew a zigzag line, the trend of this line (actually a sequence of points) would be very low (close to zero) because each zig trend would tend to cancel out each zag trend], 0332-0333 [line of the X graphic…delete]]);
determining a relationship between the multiple shapes based on a relative positioning of the single line with respect to the multiple shapes (see citations above and also see ¶¶ 0019 [context…location, proximity], 0024[rectangle (shape)…location…can affect other objects…drawing connecting lines; see with 0054 and 0063-0064 [arrows/lines…can be hand drawn to designate links (relationships) between any graphic object or group of objects (shapes) and any other graphic object or group of objects; see with 0065-0067 [example given – arrow/line…nature of the link]]]; see also 0320-0323);
generating, for display in a user interface, a structured workflow diagram including compute nodes representing the multiple shapes that are connected by a generated line representing the relationship (¶¶ 0064 [the definition(s) of arrows' functions can be determined by the graphical context of the arrows: the way the arrows are drawn, where the arrows are drawn, the span of the graphical context of the arrows, where the arrows start and where they end up, and as mentioned above, when they were drawn or when they were started and then completed. It is the graphical context which the arrow encompasses together with the user defined nature of an arrow (herein termed "arrow logics") that can define that arrow's function (workflow is described here). When one draws an arrow between certain types of graphical objects (nodes), different operations may be commanded, depending upon the context(s) of the arrow; see with 0065-0068 [example of this workflow – object recognized by this system may have its own info window that can be used for the user-specification…definitions, assignment(s), operations, functions, actions…one can link these two knobs by drawing an arrow between them. Note that the direction of the arrow between the knobs is crucial in defining the nature of the link…operation of the arrow…arrows generally represent actions that are governed by logics and as such, once an arrow is drawn and recognized and its action carried out (workflow)], 0056-0058 [user interface… graphic on a screen], 0024 [display on screen/interface]]); and
generating a structured workflow representing the structured workflow diagram (¶¶ 0038 [hand drawn graphics…used to create…path…flow…order of events, types of devices, groupings of devices, relationships between devices; with 0055+ [hand drawn…becomes…flow diagram]]).
Jaeger does not explicitly state memory (although Jaeger does disclose computing system storing information).
Analogous art Dillner discloses memory (¶¶ 0072 [stored in computer memory], 0081 [computer memory]).
Therefore, it would be obvious to one of ordinary skill in the art to include in the system/method of Jaeger memory as taught by analogous art Dillner since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Dillner would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141; and also see (1) 2007 Examination Guidelines for Determining Obviousness Under 35 U.S.C. 103 in View of the Supreme Court Decision in KSR International Co. v. Teleflex Inc. - Federal Register, Vol. 72, No. 195, October 10, 2007, pages 57526-57535; (2) 2010 Examination Guidelines Updated Developments in the Obviousness Inquiry After KSR v. Teleflex. -Federal Register, Vol. 75, No. 169, September 01, 2010, pages 53643-53660; and (3) materials posted at https://www.uspto.gov/patent/laws-and-regulations/examination-policy/examination-guidelines-training-materials-view-ksr).
Jaeger does not explicitly state file.
Analogous art Dillner discloses files and flow diagrams (¶¶ 0063 [hand drawn…results in flow diagram; with 0075 [transformed to a file that specifies…diagram; with 0209 [diagrams usually result in the construction of a directed graph that appropriately represents the flow of data in the diagram]], 0192 [representation…visual…environment…converted into a program understood by the visual programming environment (such as a text file specifying a diagram)]]).
Therefore, it would be obvious to one of ordinary skill in the art to include in the system/method of Jaeger file as taught by analogous art Dillner in order to efficiently provide essential instructions to the computing environment since doing so could be performed readily by any person of ordinary skill in the art, with neither undue experimentation, nor risk of unexpected results (KSR-G/TSM); and also since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Dillner would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141; and also see (1) 2007 Examination Guidelines for Determining Obviousness Under 35 U.S.C. 103 in View of the Supreme Court Decision in KSR International Co. v. Teleflex Inc. - Federal Register, Vol. 72, No. 195, October 10, 2007, pages 57526-57535; (2) 2010 Examination Guidelines Updated Developments in the Obviousness Inquiry After KSR v. Teleflex. -Federal Register, Vol. 75, No. 169, September 01, 2010, pages 53643-53660; and (3) materials posted at https://www.uspto.gov/patent/laws-and-regulations/examination-policy/examination-guidelines-training-materials-view-ksr).
As per claim 15, Jaeger discloses the system of claim 13, wherein detecting the multiple shapes includes detecting two or more shapes having centers that are within a threshold distance from one another and generating a merged shape by merging the two or more shapes, wherein determining the relationship includes determining the relationship between the merged shape and an additional shape of the multiple shapes (¶¶ 0206-0208 [detecting…line…shape; with 0212-0214 [determine…distance…threshold value…does not exceed the threshold…midpoint], 0041 [recognize what are various hand drawn images resides in the software code as sets of rules and as real time algorithms; with 0022 [use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects], and 0214 [a vertex determination is made if the slice angle…exceeds the current threshold angle… past the region…which is at (or close to) the midpoint…places the vertex at the midpoint]]; Jaeger’s claims 56-58 [showing workflow – joining…in a utilitarian sequence…joining step includes the step of drawing at least one line extending to and intersecting…in said sequence], se with 0091 [combination of two or more…shapes; with 0247 [example – although a rectangle or folder or triangle…combined…identification as a circle], 0328 [operates to combine other graphic objects]], 0019 [context…location, proximity], 0024 [rectangle (shape)…location…can affect other objects…drawing connecting lines; see with 0054 and 0063-0064 [arrows/lines…can be hand drawn to designate links (relationships) between any graphic object or group of objects (shapes) and any other graphic object or group of objects; see with 0065-0067 [example given – arrow/line…nature of the link]]]; see also 0320-0323).
As per claim 16, Jaeger discloses the system of claim 13, the operations further comprising extending the single line, resulting in an extended line (Jaeger’s claim 58 [joining step includes the step of drawing at least one line extending], claim 79 [line that extends]; also see ¶¶ 0403),
wherein determining the relationship is based on the relative positioning of the extended line with respect to the multiple shapes (see citations above and also see ¶¶ 0019 [context…location, proximity], 0024[rectangle (shape)…location…can affect other objects…drawing connecting lines; see with 0054 and 0063-0064 [arrows/lines…can be hand drawn to designate links (relationships) between any graphic object or group of objects (shapes) and any other graphic object or group of objects; see with 0065-0067 [example given – arrow/line…nature of the link]]]; see also 0320-0323).
As per claim 19, claim 19 discloses substantially similar limitations as claim 16 above; and therefore claim 19 is rejected under the same rationale and reasoning as presented above for claim 16.
As per claim 17, Jaeger discloses instructions that, responsive to execution by at least one processing device, cause the at least one processing device to perform operations (¶¶ 0028 [processor…hardware…software], 0093 [system…processor], 0184, 0212-0213 [system stores data], 0230-0235, 0325) including:
receiving an image of a human-drawn workflow diagram (¶¶ 0006 [recognize hand drawn objects…inputting data on…screen], 0041 [hand drawn images are recognized…input device]; claim 51 [input device to freehand draw…hand drawn entry into a functional device]);
detecting, using one or more image recognition algorithms, multiple shapes and a line in the human-drawn workflow diagram, the multiple shapes including two or more shapes having centers that are less than a threshold distance from one another (¶¶ 0206-0208 [detecting…line…shape; with 0212-0214 [determine…distance…threshold value…does not exceed the threshold…midpoint], 0041 [recognize what are various hand drawn images resides in the software code as sets of rules and as real time algorithms; with 0022 [use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects], and 0214 [a vertex determination is made if the slice angle…exceeds the current threshold angle… past the region…which is at (or close to) the midpoint…places the vertex at the midpoint]]; Jaeger’s claims 56-58 [showing workflow – joining…in a utilitarian sequence…joining step includes the step of drawing at least one line extending to and intersecting…in said sequence]);
generating a merged shape by merging the two or more shapes (¶¶ 0091 [combination of two or more…shapes; with 0247 [example – although a rectangle or folder or triangle…combined…identification as a circle], 0328 [operates to combine other graphic objects]]);
determining a relationship between the merged shape and an additional shape of the multiple shapes based on a relative positioning of the line with respect to the merged shape and the additional shape (see citations above and also see ¶¶ 0019 [context…location, proximity], 0024[rectangle (shape)…location…can affect other objects…drawing connecting lines; see with 0054 and 0063-0064 [arrows/lines…can be hand drawn to designate links (relationships) between any graphic object or group of objects (shapes) and any other graphic object or group of objects; see with 0065-0067 [example given – arrow/line…nature of the link]]]; see also 0320-0323);
generating, for display in a user interface, a structured workflow diagram including compute nodes representing the merged shape and the additional shape that are connected by a generated line representing the relationship (¶¶ 0064 [the definition(s) of arrows' functions can be determined by the graphical context of the arrows: the way the arrows are drawn, where the arrows are drawn, the span of the graphical context of the arrows, where the arrows start and where they end up, and as mentioned above, when they were drawn or when they were started and then completed. It is the graphical context which the arrow encompasses together with the user defined nature of an arrow (herein termed "arrow logics") that can define that arrow's function (workflow is described here). When one draws an arrow between certain types of graphical objects (nodes), different operations may be commanded, depending upon the context(s) of the arrow; see with 0065-0068 [example of this workflow – object recognized by this system may have its own info window that can be used for the user-specification…definitions, assignment(s), operations, functions, actions…one can link these two knobs by drawing an arrow between them. Note that the direction of the arrow between the knobs is crucial in defining the nature of the link…operation of the arrow…arrows generally represent actions that are governed by logics and as such, once an arrow is drawn and recognized and its action carried out (workflow)], 0056-0058 [user interface… graphic on a screen], 0024 [display on screen/interface]]); and
generating a structured workflow representing the structured workflow diagram (¶¶ 0038 [hand drawn graphics…used to create…path…flow…order of events, types of devices, groupings of devices, relationships between devices; with 0055+ [hand drawn…becomes…flow diagram]]).
Jaeger does not explicitly state non-transitory computer-readable storage media (although Jaeger does disclose computing system storing instruction/software and data/information).
Analogous art Dillner discloses non-transitory computer-readable storage media (¶¶ 0259 [computer-readable medium, such as an optical or magnetic disk or memory chip]).
Therefore, it would be obvious to one of ordinary skill in the art to include in the system/method of Jaeger non-transitory computer-readable storage media as taught by analogous art Dillner since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Dillner would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141; and also see (1) 2007 Examination Guidelines for Determining Obviousness Under 35 U.S.C. 103 in View of the Supreme Court Decision in KSR International Co. v. Teleflex Inc. - Federal Register, Vol. 72, No. 195, October 10, 2007, pages 57526-57535; (2) 2010 Examination Guidelines Updated Developments in the Obviousness Inquiry After KSR v. Teleflex. -Federal Register, Vol. 75, No. 169, September 01, 2010, pages 53643-53660; and (3) materials posted at https://www.uspto.gov/patent/laws-and-regulations/examination-policy/examination-guidelines-training-materials-view-ksr).
Jaeger does not explicitly state file.
Analogous art Dillner discloses files and flow diagrams (¶¶ 0063 [hand drawn…results in flow diagram; with 0075 [transformed to a file that specifies…diagram; with 0209 [diagrams usually result in the construction of a directed graph that appropriately represents the flow of data in the diagram]], 0192 [representation…visual…environment…converted into a program understood by the visual programming environment (such as a text file specifying a diagram)]]).
Therefore, it would be obvious to one of ordinary skill in the art to include in the system/method of Jaeger file as taught by analogous art Dillner in order to efficiently provide essential instructions to the computing environment since doing so could be performed readily by any person of ordinary skill in the art, with neither undue experimentation, nor risk of unexpected results (KSR-G/TSM); and also since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Dillner would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141; and also see (1) 2007 Examination Guidelines for Determining Obviousness Under 35 U.S.C. 103 in View of the Supreme Court Decision in KSR International Co. v. Teleflex Inc. - Federal Register, Vol. 72, No. 195, October 10, 2007, pages 57526-57535; (2) 2010 Examination Guidelines Updated Developments in the Obviousness Inquiry After KSR v. Teleflex. -Federal Register, Vol. 75, No. 169, September 01, 2010, pages 53643-53660; and (3) materials posted at https://www.uspto.gov/patent/laws-and-regulations/examination-policy/examination-guidelines-training-materials-view-ksr).
Claims 3-4 are rejected under 35 U.S.C. 103 as being unpatentable over Jaeger et al., (US 2002/0141643) in view of Dillner (US 2004/0090439), further in view of Wekel et al., (US 2021/0063578)
As per claim 3, Jaeger discloses the method of claim 1, and detecting the shapes and the lines within images of human-drawn objects (see citations above for claim 1 and see ¶¶ 0206-0208 [detecting…line…shape; with 0041 [recognize what are various hand drawn images resides in the software code as sets of rules and as real time algorithms; with 0022 [use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects ], 0055 [hand drawn…flow diagram]]; Jaeger’s claims 56-58 [showing workflow – joining…in a utilitarian sequence…joining step includes the step of drawing at least one line extending to and intersecting…in said sequence]).
However neither Jaeger nor Dillner disclose using a convolutional neural network having been trained.
Analogous art Wekel discloses a convolutional neural network having been trained (¶¶ 0048-0051 [image…training…convolutional neural network (CNN)]).
Therefore, it would be obvious to one of ordinary skill in the art to include in Jaeger in view of Dillner convolutional neural network having been trained as taught by analogous art Wekel in order to efficiently and optimal process images (efficiently extract spatial features) using AI/ML models since doing so could be performed readily by any person of ordinary skill in the art, with neither undue experimentation, nor risk of unexpected results (KSR-G/TSM); and also since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Wekel (using CNNs AI/ML models is well-known in imaging processing) would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141; and also see (1) 2007 Examination Guidelines for Determining Obviousness Under 35 U.S.C. 103 in View of the Supreme Court Decision in KSR International Co. v. Teleflex Inc. - Federal Register, Vol. 72, No. 195, October 10, 2007, pages 57526-57535; (2) 2010 Examination Guidelines Updated Developments in the Obviousness Inquiry After KSR v. Teleflex. -Federal Register, Vol. 75, No. 169, September 01, 2010, pages 53643-53660; and (3) materials posted at https://www.uspto.gov/patent/laws-and-regulations/examination-policy/examination-guidelines-training-materials-view-ksr).
As per claim 4, Jaeger discloses the method of claim 1, wherein detecting the shapes and the lines includes detecting the shapes and the lines in the human-drawn workflow diagram using circles and lines detection/recognition software on devices (see citations for claim 1 above and see ¶¶ 0006 [recognize hand drawn objects…inputting data on…screen], 0041 [hand drawn images are recognized…input device]; claim 51 [input device to freehand draw…hand drawn entry into a functional device], 0206-0208 [detecting…line…shape; with 0041 [recognize what are various hand drawn images resides in the software code as sets of rules and as real time algorithms; with 0022 [use of "arrow logics" and "line logics" for programming connections, functions, links and behaviors and many types of associations between hand drawn objects ], 0055 [hand drawn…flow diagram]]; Jaeger’s claims 56-58 [showing workflow – joining…in a utilitarian sequence…joining step includes the step of drawing at least one line extending to and intersecting…in said sequence]).
However, neither Jaeger nor Dillner state using Hough Transforms.
Analogous art Wekel discloses Hough Transforms (¶¶ 0043 [image…geometric…Hough transform algorithm…applied]).
Therefore, it would be obvious to one of ordinary skill in the art to include in Jaeger in view of Dillner Hough Transforms as taught by analogous art Wekel in order to efficiently and optimal process images (efficiently and accurately detect shapes in digital images) since doing so could be performed readily by any person of ordinary skill in the art, with neither undue experimentation, nor risk of unexpected results (KSR-G/TSM); and also since one of ordinary skill in the art at the time of the invention would have recognized that applying the known technique and concepts of Wekel (Hough transforms is well-known in imaging processing for detecting shapes) would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such concepts and features into similar systems (KSR-D). (MPEP 2141; and also see (1) 2007 Examination Guidelines for Determining Obviousness Under 35 U.S.C. 103 in View of the Supreme Court Decision in KSR International Co. v. Teleflex Inc. - Federal Register, Vol. 72, No. 195, October 10, 2007, pages 57526-57535; (2) 2010 Examination Guidelines Updated Developments in the Obviousness Inquiry After KSR v. Teleflex. -Federal Register, Vol. 75, No. 169, September 01, 2010, pages 53643-53660; and (3) materials posted at https://www.uspto.gov/patent/laws-and-regulations/examination-policy/examination-guidelines-training-materials-view-ksr).
Conclusion
The prior art made of record on the PTO-892 and not relied upon is considered pertinent to applicant's disclosure. For example, some of the pertinent prior art is as follows:
Murakami (US 2003/0123698): Provides an image processing apparatus and method, which embed a digital watermark having strong robustness against geometrical transformation/edit processes such as rotation, scaling, translation, and the like, and an image processing apparatus and method, which can accurately extract the embedded digital watermark even after the conversion/edit process such as rotation of an image or the like. A Fourier transformer (901) converts an image input from an image input unit (900) into amplitude component data, and phase component data. An envelope ring pattern generator (902) generates an embedding pattern of digital watermark information, which is expressed by a set of lines that define a circle as an envelope, on the amplitude component data. An envelope ring pattern embedding unit (903) embeds digital watermark information at predetermined positions on the line.
Wang et al., (US 2014/0328544): Discusses a sketch-based image recognition system that may generate a model for identifying a subject of a sketch. The model is formed from a plurality of images having visual features similar to the visual features of the sketch. The model may include object topics representative of categories which may correspond to the subject of the sketch and shape topics representative of the visual features of the sketch.
Schaefer et al., (US 2023/0062484): Provides for converting a diagram into a digital diagram format. A diagram is received as an image file and a plurality of recognition stages produce a final diagram based on predicted information from one or more of the recognition stages. The plurality of recognition stages including a shape detection stage for detecting a plurality of shapes and at least one arrow detection stage in which arrows are detected as relations between pairs of shapes. The final diagram is generated based on the predicted information and is converted into the digital diagram format compatible with a diagram modeling language.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GURKANWALJIT SINGH whose telephone number is (571)270-5392. The examiner can normally be reached on M-F 8:30-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Brian Epstein can be reached on 571-270-5389. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Gurkanwaljit Singh/
Primary Examiner, Art Unit 3625
Prosecution Insights