DETAILED ACTION
The action is responsive to the Application filed on 03/30/2023. Claims 1-15 are pending in the case. Claims 1, 7 and 13-15 are independent claims.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant's claim for foreign priority based on an application filed in Europe on 04/06/2022. It is noted, however, that applicant has not filed a certified copy of the EP22166871.8 application as required by 37 CFR 1.55.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 5 and 11 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. The claims recite “the annotation extracting methods” which lacks antecedent basis therefore making the claim indefinite. For the purposes of examination, Examiner assumed the claims to recite “[[the ]]annotation extracting methods”.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 15 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. During examination, the claims must be interpreted as broadly as their terms reasonably allow (In re American Academy of Science Tech Center, 367 F.3d 1359, 1369, 70 U.S.P.Q.2d 1827, 1834 (Fed. Cir. 2004)). Claim 15 recites "a computer readable hardware storage device," which is not comprehensively defined by the specification. The broadest reasonable interpretation of a claim drawn to a computer readable hardware storage device covers forms of transitory propagating signals per se in view of the ordinary and customary meaning of computer readable media, particularly when the specification fails to expressly exclude them. Transitory propagating signals are non-statutory subject matter. (In re Nuijten, 500 F.3d 1346, 1356-57, 84 U.S.P.Q.2d 1495, 1502 (Fed. Cir. 2007)) (transitory embodiments are not directed to statutory subject matter). See also Subject Matter Eligibility of Computer Readable Media (1351 Off. Gaz. Pat. Office 212 (Feb. 23, 2010)). Examiner suggests adding the word "non-transitory." Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 3, 5-7, 9 and 11-15 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Mayer et al. (US 20230108015 A1, hereinafter Mayer).
As to claim 1, Mayer discloses a computer-implemented method for generating a configuration for external datapoint access, whereby the configuration comprises at least one datapoint within an automation system ("Robots 130 in some embodiments are split into several components, each being dedicated to a particular automation task. The robot components in some embodiments include, but are not limited to, SCM-managed robot services, user mode robot services, executors, agents, and command line," Mayer paragraph 0043; "Some embodiments bring semantic automation into automation platforms for creating fully automated workflows with less or minimal interaction input from the developer. Using semantic mapping, the UI fields from a data source/source screen are mapped to UI fields on the target screen semantically using one or more AI/ML models, and fully automated workflows can be created from this semantic mapping without intervention by the developer," Mayer paragraph 0102; "Source data other than images can also be used in some embodiments. For example, when the developer selects map data option 914 and clicks select source button 926 in FIG. 9D, data source options 927 appear. See FIG. 9E. The developer can select the desired source data format, such as Excel®, JavaScript® Object Notation (JSON), XAML of an RPA workflow, a comma separated variable (CSV) file, etc.," Mayer paragraph 0110, source data (i.e., a datapoint) is used in a automated workflow (i.e., an automation system)), the method comprising the following method steps which can be executed by one or more processors:
a) searching for and capturing at least one input and/or output field, named an I/O-field, as an element of several elements on at least one user interface surface ("When the developer selects map screens option 912, a select source button 922 and a select target button 924 appear in mapping pane 920. The user can return to the previous designer application screen by clicking back button 930. When the user clicks one of these buttons, the user can select the source and target using indicate on screen functionality similar to or the same as that of UiPath Studio™ in some embodiments. See, for example, U.S. patent application Ser. No. 17/100,146. This causes a selected source screen 940 and a selected target screen 950 to be displayed in mapping pane 920. It should be noted that source screen 940 and/or target screen 950 may be application windows, portions of displayed applications, etc.," Mayer paragraph 0104; "When the user clicks map button 932, the designer application calls one or more AI/ML models that perform OCR and CV on source screen 940 and target screen 950, runs semantic AI analysis attempting to match fields in source screen 940 with those of target screen 950, and displays matches with confidence scores that meet or exceed a confidence threshold. See FIG. 9B. A global confidence score 960 is also displayed," Mayer paragraph 0105);
b) extracting annotation data near to a visualized automation process value in a found I/O-field and surrounding the I/O-field ("For instance, because the field City appears directly to the left of it associated text field in target screen 950 and no other text field includes this label, the designer application and/or AI/ML model(s) may determine that these fields are linked, and assign the City label as an anchor for the target text field," Mayer paragraph 0112);
c) attributing extracted annotation data to at least one datapoint within the automation system, whereby each datapoint is related to supplemental information of corresponding engineering project artifacts of the automation system ("Source data other than images can also be used in some embodiments. For example, when the developer selects map data option 914 and clicks select source button 926 in FIG. 9D, data source options 927 appear. See FIG. 9E. The developer can select the desired source data format, such as Excel®, JavaScript® Object Notation (JSON), XAML of an RPA workflow, a comma separated variable (CSV) file, etc.," Mayer paragraph 0110; "When the user clicks map button 932, the designer application calls one or more AI/ML models that perform OCR and CV on source screen 940 and target screen 950, runs semantic AI analysis attempting to match fields in source screen 940 with those of target screen 950, and displays matches with confidence scores that meet or exceed a confidence threshold. See FIG. 9B. A global confidence score 960 is also displayed," Mayer paragraph 0105, supplemental data that shows where to get the data from);
d) providing a data scheme built with via edges linked components representing elements of the user interface surface, the exacted annotation data and the at least one datapoint as well as the supplemental information, wherein each edge represents a relationship between two components ("After the source screen or source data and the target screen have been mapped, the user can click Create button 934 to automatically generate one or more activities in the RPA workflow that implement the desired mapping. See FIGS. 9C and 9G. This causes the RPA workflow activities to be automatically created. In some embodiments, the RPA workflow is immediately executed to perform the mapping task desired by the user after creation," Mayer paragraph 0113, RPA workflow scheme that correlates datapoint, label data and supplemental information);
e) querying a search through the data scheme for one or more visualized pre-selected automation process values ("Such an example is shown in FIG. 10, which illustrates an RPA designer application 1000 with automatically generated activities in an RPA workflow 1010, according to an embodiment of the present invention. The semantic matching AI/ML model(s) have been trained to recognize associations between the source screen or source data and the target screen, per the above. In the case of the example of FIGS. 9A-G and 10, the semantic matching AI/ML model(s) are able to determine that data from fields in the source screen or source data should be copied into the matching fields in the target screen. Accordingly, RPA designer application 1000 knows to obtain UI descriptors for the target elements from the UI object repository, add activities to RPA workflow 1010 that click on the target screen, click on each target field, and enter the text from the source screen or data source into the respective matching fields in the target screen using these UI descriptors. RPA designer application 1000 automatically generates one or more activities in RPA workflow 1010 that implement this functionality," Mayer paragraph 0115); and
f) generating a configuration with at least one datapoint along with linked supplemental information as a result from the search, wherein the linked supplemental information comprises information how to externally access the at least one datapoint ("After the source screen or source data and the target screen have been mapped, the user can click Create button 934 to automatically generate one or more activities in the RPA workflow that implement the desired mapping. See FIGS. 9C and 9G. This causes the RPA workflow activities to be automatically created. In some embodiments, the RPA workflow is immediately executed to perform the mapping task desired by the user after creation," Mayer paragraph 0113); and
g) outputting the configuration in an automation system readable and/or computer-readable format ("After the source screen or source data and the target screen have been mapped, the user can click Create button 934 to automatically generate one or more activities in the RPA workflow that implement the desired mapping. See FIGS. 9C and 9G. This causes the RPA workflow activities to be automatically created. In some embodiments, the RPA workflow is immediately executed to perform the mapping task desired by the user after creation," Mayer paragraph 0113; "Such an example is shown in FIG. 10, which illustrates an RPA designer application 1000 with automatically generated activities in an RPA workflow 1010, according to an embodiment of the present invention. The semantic matching AI/ML model(s) have been trained to recognize associations between the source screen or source data and the target screen, per the above. In the case of the example of FIGS. 9A-G and 10, the semantic matching AI/ML model(s) are able to determine that data from fields in the source screen or source data should be copied into the matching fields in the target screen. Accordingly, RPA designer application 1000 knows to obtain UI descriptors for the target elements from the UI object repository, add activities to RPA workflow 1010 that click on the target screen, click on each target field, and enter the text from the source screen or data source into the respective matching fields in the target screen using these UI descriptors. RPA designer application 1000 automatically generates one or more activities in RPA workflow 1010 that implement this functionality," Mayer paragraph 0115, RPA workflow can be executed (i.e., is in a computer readable format)).
As to claim 3, Mayer further discloses the method according to claim 1, wherein annotation data is extracted by scanning annotation typical regions surrounding the I/O-field ("For instance, because the field City appears directly to the left of it associated text field in target screen 950 and no other text field includes this label, the designer application and/or AI/ML model(s) may determine that these fields are linked, and assign the City label as an anchor for the target text field," Mayer paragraph 0112).
As to claim 5, Mayer further discloses the method according to claim 1, wherein annotation extracting methods are combined with each other and each of them are weighted, whereby weighting is introduced into an optimization method or is learned via machine learning in order to reach a minimal cost assignment to the weighting ("During training, various labeled data (in this case, images) are fed through neural network 700. Successful identifications strengthen weights for inputs to neurons, whereas unsuccessful identifications weaken them. A cost function, such as mean square error (MSE) or gradient descent may be used to punish predictions that are slightly wrong much less than predictions that are very wrong. If the performance of the AI/ML model is not improving after a certain number of training iterations, a data scientist may modify the reward function, provide indications of where non-identified graphical elements are, provide corrections of misidentified graphical elements, etc.," Mayer paragraph 0088).
As to claim 6, Mayer further discloses the method according to claim 1, wherein extracted annotation data is attributed to a visualized automation process value by matching the annotation string against a string of the at least one datapoint with a certain degree of differences ("When the user clicks map button 932, the designer application calls one or more AI/ML models that perform OCR and CV on source screen 940 and target screen 950, runs semantic AI analysis attempting to match fields in source screen 940 with those of target screen 950, and displays matches with confidence scores that meet or exceed a confidence threshold. See FIG. 9B. A global confidence score 960 is also displayed," Mayer paragraph 0105; "Source data other than images can also be used in some embodiments. For example, when the developer selects map data option 914 and clicks select source button 926 in FIG. 9D, data source options 927 appear. See FIG. 9E. The developer can select the desired source data format, such as Excel®, JavaScript® Object Notation (JSON), XAML of an RPA workflow, a comma separated variable (CSV) file, etc.," Mayer paragraph 0110; "Relationships between labels in the source screen and target screen may be used to determine what a given text field is meant to represent, although the text fields may be similar to or the same as one another. This may be accomplished by assigning one or more anchors to a given text field. For instance, because the field City appears directly to the left of it associated text field in target screen 950 and no other text field includes this label, the designer application and/or AI/ML model(s) may determine that these fields are linked, and assign the City label as an anchor for the target text field," Mayer paragraph 0112).
As to claim 7, Mayer discloses a data processing system for generating a configuration for external datapoint access, whereby the configuration comprises at least one datapoint within an automation system ("Robots 130 in some embodiments are split into several components, each being dedicated to a particular automation task. The robot components in some embodiments include, but are not limited to, SCM-managed robot services, user mode robot services, executors, agents, and command line," Mayer paragraph 0043; "Some embodiments bring semantic automation into automation platforms for creating fully automated workflows with less or minimal interaction input from the developer. Using semantic mapping, the UI fields from a data source/source screen are mapped to UI fields on the target screen semantically using one or more AI/ML models, and fully automated workflows can be created from this semantic mapping without intervention by the developer," Mayer paragraph 0102; "Source data other than images can also be used in some embodiments. For example, when the developer selects map data option 914 and clicks select source button 926 in FIG. 9D, data source options 927 appear. See FIG. 9E. The developer can select the desired source data format, such as Excel®, JavaScript® Object Notation (JSON), XAML of an RPA workflow, a comma separated variable (CSV) file, etc.," Mayer paragraph 0110, source data (i.e., a datapoint) is used in a automated workflow (i.e., an automation system)), whereby the system comprises one or more processors (“The process steps performed in FIGS. 13 and 14 may be performed by a computer program, encoding instructions for the processor(s) to perform at least part of the process(es) described in FIGS. 13 and 14, in accordance with embodiments of the present invention,” Mayer paragraph 0135) which is or are configured to:
a) search for and capture at least one input and/or output field, named an I/O-field, as an element of several elements on at least one user interface surface ("When the developer selects map screens option 912, a select source button 922 and a select target button 924 appear in mapping pane 920. The user can return to the previous designer application screen by clicking back button 930. When the user clicks one of these buttons, the user can select the source and target using indicate on screen functionality similar to or the same as that of UiPath Studio™ in some embodiments. See, for example, U.S. patent application Ser. No. 17/100,146. This causes a selected source screen 940 and a selected target screen 950 to be displayed in mapping pane 920. It should be noted that source screen 940 and/or target screen 950 may be application windows, portions of displayed applications, etc.," Mayer paragraph 0104; "When the user clicks map button 932, the designer application calls one or more AI/ML models that perform OCR and CV on source screen 940 and target screen 950, runs semantic AI analysis attempting to match fields in source screen 940 with those of target screen 950, and displays matches with confidence scores that meet or exceed a confidence threshold. See FIG. 9B. A global confidence score 960 is also displayed," Mayer paragraph 0105);
b) extract annotation data near to a visualized automation process value in a found I/O-field and surrounding the I/O-field ("For instance, because the field City appears directly to the left of it associated text field in target screen 950 and no other text field includes this label, the designer application and/or AI/ML model(s) may determine that these fields are linked, and assign the City label as an anchor for the target text field," Mayer paragraph 0112);
c) attribute extracted annotation data to at least one datapoint within the automation system, whereby each datapoint is related to supplemental information of corresponding engineering project artifacts of the automation system ("Source data other than images can also be used in some embodiments. For example, when the developer selects map data option 914 and clicks select source button 926 in FIG. 9D, data source options 927 appear. See FIG. 9E. The developer can select the desired source data format, such as Excel®, JavaScript® Object Notation (JSON), XAML of an RPA workflow, a comma separated variable (CSV) file, etc.," Mayer paragraph 0110; "When the user clicks map button 932, the designer application calls one or more AI/ML models that perform OCR and CV on source screen 940 and target screen 950, runs semantic AI analysis attempting to match fields in source screen 940 with those of target screen 950, and displays matches with confidence scores that meet or exceed a confidence threshold. See FIG. 9B. A global confidence score 960 is also displayed," Mayer paragraph 0105, supplemental data that shows where to get the data from);
d) provide a data scheme built with via edges linked components representing elements of the user interface surface, the exacted annotation data and the at least one datapoint as well as the supplemental information, wherein each edge represents a relationship between two components ("After the source screen or source data and the target screen have been mapped, the user can click Create button 934 to automatically generate one or more activities in the RPA workflow that implement the desired mapping. See FIGS. 9C and 9G. This causes the RPA workflow activities to be automatically created. In some embodiments, the RPA workflow is immediately executed to perform the mapping task desired by the user after creation," Mayer paragraph 0113, RPA workflow scheme that correlates datapoint, label data and supplemental information);
e) query a search through the data scheme for one or more visualized automation process values ("Such an example is shown in FIG. 10, which illustrates an RPA designer application 1000 with automatically generated activities in an RPA workflow 1010, according to an embodiment of the present invention. The semantic matching AI/ML model(s) have been trained to recognize associations between the source screen or source data and the target screen, per the above. In the case of the example of FIGS. 9A-G and 10, the semantic matching AI/ML model(s) are able to determine that data from fields in the source screen or source data should be copied into the matching fields in the target screen. Accordingly, RPA designer application 1000 knows to obtain UI descriptors for the target elements from the UI object repository, add activities to RPA workflow 1010 that click on the target screen, click on each target field, and enter the text from the source screen or data source into the respective matching fields in the target screen using these UI descriptors. RPA designer application 1000 automatically generates one or more activities in RPA workflow 1010 that implement this functionality," Mayer paragraph 0115); and
f) generate a configuration with at least one datapoint along with linked supplemental information as a result from the search, wherein the linked supplemental information comprises information how to externally access the at least one datapoint ("After the source screen or source data and the target screen have been mapped, the user can click Create button 934 to automatically generate one or more activities in the RPA workflow that implement the desired mapping. See FIGS. 9C and 9G. This causes the RPA workflow activities to be automatically created. In some embodiments, the RPA workflow is immediately executed to perform the mapping task desired by the user after creation," Mayer paragraph 0113); and
g) output the configuration in an automation system readable and/or computer-readable format ("After the source screen or source data and the target screen have been mapped, the user can click Create button 934 to automatically generate one or more activities in the RPA workflow that implement the desired mapping. See FIGS. 9C and 9G. This causes the RPA workflow activities to be automatically created. In some embodiments, the RPA workflow is immediately executed to perform the mapping task desired by the user after creation," Mayer paragraph 0113; "Such an example is shown in FIG. 10, which illustrates an RPA designer application 1000 with automatically generated activities in an RPA workflow 1010, according to an embodiment of the present invention. The semantic matching AI/ML model(s) have been trained to recognize associations between the source screen or source data and the target screen, per the above. In the case of the example of FIGS. 9A-G and 10, the semantic matching AI/ML model(s) are able to determine that data from fields in the source screen or source data should be copied into the matching fields in the target screen. Accordingly, RPA designer application 1000 knows to obtain UI descriptors for the target elements from the UI object repository, add activities to RPA workflow 1010 that click on the target screen, click on each target field, and enter the text from the source screen or data source into the respective matching fields in the target screen using these UI descriptors. RPA designer application 1000 automatically generates one or more activities in RPA workflow 1010 that implement this functionality," Mayer paragraph 0115, RPA workflow can be executed (i.e., is in a computer readable format)).
As to claim 9, it is substantially similar to claim 3 and is therefore rejected using the same rationale as above.
As to claim 11, it is substantially similar to claim 5 and is therefore rejected using the same rationale as above.
As to claim 12, it is substantially similar to claim 6 and is therefore rejected using the same rationale as above.
As to claim 13, Mayer discloses a device having a processor and/or controller (“The process steps performed in FIGS. 13 and 14 may be performed by a computer program, encoding instructions for the processor(s) to perform at least part of the process(es) described in FIGS. 13 and 14, in accordance with embodiments of the present invention,” Mayer paragraph 0135) which is configured to:
receive a generated configuration with at least one datapoint along with linked supplemental information as a result from a search through a data scheme for one or more visualized automation process values, wherein the linked supplemental information comprises information how to externally access the at least one datapoint (“After the source screen or source data and the target screen have been mapped, the user can click Create button 934 to automatically generate one or more activities in the RPA workflow that implement the desired mapping. See FIGS. 9C and 9G. This causes the RPA workflow activities to be automatically created. In some embodiments, the RPA workflow is immediately executed to perform the mapping task desired by the user after creation,” Mayer paragraph 0113; “Accordingly, RPA designer application 1000 knows to obtain UI descriptors for the target elements from the UI object repository, add activities to RPA workflow 1010 that click on the target screen, click on each target field, and enter the text from the source screen or data source into the respective matching fields in the target screen using these UI descriptors. RPA designer application 1000 automatically generates one or more activities in RPA workflow 1010 that implement this functionality. In some embodiments, the developer may not be permitted to modify these activities,” Mayer paragraph 0115, receiving and executing the RPA workflow script that has the mapping between datapoint source (i.e., supplemental information), datapoint values and UI elements and labels);
access the at least one datapoint for analyzing and/or measuring and/or monitoring their automation process values (“Accordingly, RPA designer application 1000 knows to obtain UI descriptors for the target elements from the UI object repository, add activities to RPA workflow 1010 that click on the target screen, click on each target field, and enter the text from the source screen or data source into the respective matching fields in the target screen using these UI descriptors. RPA designer application 1000 automatically generates one or more activities in RPA workflow 1010 that implement this functionality. In some embodiments, the developer may not be permitted to modify these activities,” Mayer paragraph 0115, accessing the data source and getting the process values for use in the RAP workflow);
steer and/or control an automation system according to results from the analysis, measurement and/or monitoring (“Accordingly, RPA designer application 1000 knows to obtain UI descriptors for the target elements from the UI object repository, add activities to RPA workflow 1010 that click on the target screen, click on each target field, and enter the text from the source screen or data source into the respective matching fields in the target screen using these UI descriptors. RPA designer application 1000 automatically generates one or more activities in RPA workflow 1010 that implement this functionality. In some embodiments, the developer may not be permitted to modify these activities,” Mayer paragraph 0115, controlling the RPA workflow automation system to input data into fields).
As to claim 14, it is substantially similar to claim 7 and is therefore rejected using the same rationale as above.
As to claim 15, it is substantially similar to claim 7 and is therefore rejected using the same rationale as above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Mayer et al. (US 20230108015 A1, hereinafter Mayer) in view of Voicu (US 20210019157 A1).
As to claim 2, Mayer discloses the method according to claim 1, however Mayer does not appear to explicitly disclose a limitation wherein annotation data is extracted by discovering an annotation typical character.
Voicu teaches a limitation wherein annotation data is extracted by discovering an annotation typical character ("Robot 206 may also send captured images or screenshots of a target area to an OCR module or engine 204 to detect text or text fields, as understood by one of ordinary skill in the art, for assistance with determining relationships of a captured UI image or screen. A text field may comprise one or more text tokens. A text token may comprise one or more characters found between a set of pre-determined delimiters such as white space, punctuation characters, special characters, or the like. Text tokens may also include a number, a date, an email address, a uniform resource identifier (URI), a zip code, or the like," Voicu paragraph 0040, using special character delimiters to determine relationships between elements and text).
Accordingly it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Mayer to extract annotation data by discovering annotation special characters as taught by Voicu. One would have been motivated to make such a combination so that the system could consider more kinds of information when determining annotation data and its relationship to an element, thus resulting in an annotation data extracting mechanism that is more robust and thus enhancing the utility of the finished product.
As to claim 8, it is substantially similar to claim 2 and is therefore rejected using the same rationale as above.
Claims 4 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Mayer et al. (US 20230108015 A1, hereinafter Mayer) in view of Dunn et al. (US 20210303342 A1, hereinafter Dunn).
As to claim 4, Mayer discloses the method according to claim 1, however Mayer does not appear to explicitly disclose a limitation wherein annotation data is extracted by comparing font size of text surrounding the I/O-field with the font size of the text or value in the I/O-field or with median font size used on the at least one user interface.
Dunn teaches a limitation wherein annotation data is extracted by comparing font size of text surrounding the I/O-field with the font size of the text or value in the I/O-field or with median font size used on the at least one user interface ("As depicted in FIG. 4, an image of the display may be obtained at 402 for example, where the image is received as a screenshot or other image based representation of the user interface. In some examples, an OCR process may be performed on the acquired image such that text displayed at the user interface may be recognized and a position of the text may be identified. In some user interfaces, an entity displayed as part of the user interface may be determined based on characteristics of the entity. For example, text displayed near the top of user interface 302 may be determined to be a title element 408; such a hierarchical location amongst one or more elements in the user interface 302 may impart a meaning to such element. That is, the position and size of the text in the title element 408 in the user interface 302 may indicate that such text is a title or maintains some other importance among elements in the user interface. As another example, text element 410 displayed near an input element 412 may be classified as a prompt based on a location of the text element 410 with respect to the input element 412. The input element 412 may be identified as an input element based on size, location, characterization (e.g., rectangle), text color with respect to the color of other text or the color of other elements, or other characteristics of the entity," Dunn paragraph 0031, using text size for correlating elements and determining importance / hierarchy).
Accordingly it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Mayer to extract annotation data by comparing font size as taught by Dunn. One would have been motivated to make such a combination so that the system could consider more kinds of information when determining annotation data and its relationship to an element, thus resulting in an annotation data extracting mechanism that is more robust and thus enhancing the utility of the finished product.
As to claim 10, it is substantially similar to claim 4 and is therefore rejected using the same rationale as above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
US 20050149552 A1 to Chan et al. discloses a method of generating data servers for heterogeneous data sources where datapoints in a source dataset are mapped to annotation elements in a target XML schema; and
US 20210133349 A1 to Jensen et al. discloses a unified data fabric for managing data lifecycles and data flows where labels of datapoints in a data source are mapped to a standard data element label.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL SAMWEL whose telephone number is (313) 446-6549. The examiner can normally be reached Monday through Thursday 8:00-6:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vu Kieu can be reached at (571) 272-4057. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL SAMWEL/Primary Examiner, Art Unit 2171