DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
2. This action is in response to the following communication: Non-provisional Application No. 18/790,393 filed on 07/31/2024.
3. Claims 1-23 are pending.
Claims 1, 22 and 23 are independent claims.
Specification
4. The disclosure is objected to because of the following informalities: The disclosure consists of abbreviations which are not written out the first time they are used (e.g. png, jpg, jpeg, tiff, CSV, LLM). Abbreviations must be written out the first time they are used in the disclosure, again in the abstract, and again in the claims, as the intent of their meaning is likely to be changed over time.
Appropriate correction is required. The specification should be revised carefully in order to comply with 35 U.S.C. 112(a). 35 U.S.C. 132(a) states that no amendment shall introduce new matter into the disclosure of the invention. Any amendment to the disclosure must be supported by the disclosure as originally filed.
Claim Rejections - 35 USC § 101
5. 35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
6. Claims 1-23 are rejected under 35 U.S.C. 101 because the claimed invention recites a judicial exception, is directed to that judicial exception, an abstract idea, as it has not been integrated into practical application and the claims further do not recite significantly more than the judicial exception. Examiner has evaluated the claims under the framework provided in the 2019 Patent Eligibility Guidance published in the Federal Register 01/07/2019 and has provided such analysis below.
Regarding claims 1, 22 and 23, the limitations “identify the user interface components and associated properties” as drafted, are functions that, under its broadest reasonable interpretation, recite the abstract idea of a mental process. These limitations encompass a human mind carrying out these functions through observation, evaluation judgment and /or opinion, or even with the aid of pen and paper. Thus, this limitation recites and falls within the “Mental Processes” grouping of abstract ideas under Prong 1.
Claims 1, 22 and 23: Under Prong 2 Step 2A, the judicial exception is not integrated into a practical application. The additional elements “a processor”, “computer program product” and “computer readable medium” merely recite instructions to implement an abstract idea on a generic computer, or merely uses a generic computer or computer components as a tool to perform the abstract idea, thus is not a practical application under Prong 2. The additional element “receiving an input”, “generate program code“, and “provide the processor with instructions“ do nothing more than add insignificant extra solution activity to the judicial exception of merely gathering data. Accordingly, the additional elements do not integrate the recited judicial exception into a practical application and the claim is therefore directed to the judicial exception. See MPEP 2106.05(f) and (g), respectively.
Claims 1, 22 and 23: Under Step 2B, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As stated above in prong 2, the additional elements “a processor”, “computer program product” and “computer readable medium” merely recite instructions to implement an abstract idea on a generic computer, or merely uses a generic computer or computer components as a tool to perform the abstract idea, and the additional element “receiving an input”, “generate program code“, and “provide the processor with instructions“ is merely gathering data which the courts have identified as well-understood, routine conventional activity. See for example Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362, MPEP 2106.05(d). Therefore, the additional elements do not amount to significantly more, thus, cannot provide an inventive concept. Accordingly, the claims are not patent eligible under 35 USC 101.
Claims 1, 22 and 23recite further additional elements “a processor”, “computer program product” and “computer readable medium”. These additional elements are recited at a high-level of generality such that it amounts no more than mere instructions to apply the exception using generic computer, and/or generic computer components. See MPEP 2106.05(f). Therefore, the additional elements recited in claims 1, 22 and 23 do not integrate the judicial exception into a practical application under prong 2, nor amount to significantly more under step 2B.
Regarding claims 2-5 and 17-21, the additional elements of “the input includes an image”, “remove noise from the input”, “identify in the input one or more shapes”, “shapes is a rectangle”, “identified shape is a repetitive shape”, “generating a two-dimensional grid structure”, “generating a corresponding token”, “generated token for the one or more identified shapes is specified”, “program code is generated by a compiler”, “generating the application program”, and “utilizing a large language model” is analyzed under Prong 2 as mere data gathering which does not integrate the judicial exception into a practical application, or amounts to significantly more under Step 2B for the reasons provided in the rejection of claims 1, 22 and 23.
Regarding claims 6 and 14 the limitation “identify the user interface components” and “determine a corresponding function” recites additional mental process under Prong 1. The additional element “loading the input”, “converting the input”, “extracted textual information is utilized“ and “applying an edge detection algorithm” is analyzed under Prong 2 as mere data gathering which does not integrate the judicial exception into a practical application, or amounts to significantly more under Step 2B for the reasons provided in the rejection of claims 1, 22 and 23.
Regarding claims 7-9, 11-13 and 15, the limitations recited in these claims merely describe the “identify the user interface components”, “determining whether any of the one or more identified shapes include one or more nested shapes”, “filtering an identified shape”, “identified shape is filtered”, “identified shape is determined”, “redefining an identified shape”, “identify the user interface… by extracting textual information”, “extracted textual information is utilized” and “determining corresponding zones associated” in each of claims 1, 22 and 23, thus, are likewise analyzed under Prong 1 as mental process.
Claim Rejections - 35 USC § 102
7. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
8. Claims 1, 2, 4, 5, 7, 8, 15, 16 and 21-23 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Liu et al., US 20210011592 (hereinafter Liu).
In regards to claim 1, Liu teaches:
A method, comprising: receiving an input specifying a schematic of user interface components of an application program (Abstract), see “a computer-implemented method, system and computer program product for generating a user interface. A sketch (e.g., wireframe) of a portion of a user interface is received”.
using a first group of one or more machine learning models to automatically identify the user interface components and associated properties specified in the input (p. 1, [0007]), see “the method additionally comprises generating and displaying a first set of intended final sketch renderings of the user interface using the first set of predicted intended sketches of the user interface based on historical data or a model trained to extract visual characteristics from existing user interface screens”.
based on the identified user interface components and the associated properties, using a second group of one or more machine learning models to automatically generate program code implementing the application program including the user interface components (p. 5, [0050]), see “in one embodiment, the model of the present invention is an encoder/decoder model that is trained on information from user interface metadata and screenshots of the user interfaces to translate the screenshots into a domain specific language and then into code”.
In regards to claim 2, Liu teaches:
the input includes an image, a prompt, a data file, or a document (Abstract), see “a computer-implemented method, system and computer program product for generating a user interface. A sketch (e.g., wireframe) of a portion of a user interface is received”.
In regards to claim 4, Liu teaches:
the one or more machine learning models of the first group identify in the input one or more shapes that correspond to the user interface components of the application program (Fig. 3, Fig. 4) and (p. 5, [0047]), see “as shown in FIG. 4, computing device 100 generates and displays a set of intended final sketch renderings 401 of the user interface using the set of predicted intended sketches 302. An “intended final sketch rendering,” as used herein, refers to a complete rendering of a user interface”.
In regards to claim 5, Liu teaches:
at least one of the one or more shapes is a rectangle (Fig. 3, Fig. 4) and (p. 5, [0047]), see “as shown in FIG. 4, computing device 100 generates and displays a set of intended final sketch renderings 401 of the user interface using the set of predicted intended sketches 302. An “intended final sketch rendering,” as used herein, refers to a complete rendering of a user interface”.
In regards to claim 7, Liu teaches:
the one or more machine learning models of the first group automatically identify the user interface components and associated properties specified in the input at least in part by determining whether any of the one or more identified shapes include one or more nested shapes (p. 5, [0054]), see “in one embodiment, a wireframe is represented in a tree-like structure, where the branches of the tree structure are container layouts and the nested user interface elements are the nodes/leaves. Matching may then be reduced to identifying two trees with the most similar branches and leaves”.
In regards to claim 8, Liu teaches:
the one or more machine learning models of the first group automatically identify the user interface components and associated properties specified in the input at least in part by filtering an identified shape of the one or more identified shapes (p. 4, [0041]), see “in one embodiment, the sketch is a wireframe. In one embodiment, in order for the wireframe to become a screen rendering (e.g., a high-fidelity digital user interface), styles (e.g., color, contrast, shape, padding, proportion) need to be applied. Such styles may be produced by a model trained to extract visual characteristics from existing user interface screens as discussed further below”.
In regards to claim 15, Liu teaches:
using the one or more machine learning models of the second group at least in part includes determining corresponding zones associated with the one or more identified shapes (Fig. 3, Fig. 4) and (p. 5, [0047], see “as shown in FIG. 4, computing device 100 generates and displays a set of intended final sketch renderings 401 of the user interface using the set of predicted intended sketches 302. An “intended final sketch rendering,” as used herein, refers to a complete rendering of a user interface”.
In regards to claim 16, Liu teaches:
using the one or more machine learning models of the second group at least in part includes generating a two-dimensional grid structure that organizes the one or more identified shapes based on the determined corresponding zones associated with the one or more identified shapes (Fig. 3, Fig. 4) and (p. 5, [0047], see “as shown in FIG. 4, computing device 100 generates and displays a set of intended final sketch renderings 401 of the user interface using the set of predicted intended sketches 302. An “intended final sketch rendering,” as used herein, refers to a complete rendering of a user interface”.
In regards to claim 21, Liu teaches:
utilizing a large language model to provide one or more insights into data associated with the application program (p. 5, [0050-0051], see “in one embodiment, the model of the present invention is an encoder/decoder model that is trained on information from user interface metadata and screenshots of the user interfaces to translate the screenshots into a domain specific language and then into code. In one embodiment, a classifier may be utilized to detect and classify user interface components within the image (image of the user interface). After classification of the interface components, the model is trained on information from the component metadata and the screenshots of the user interface to translate the screenshots into a domain specific language and then into code”.
In regards to claim 22, Liu teaches:
A system, comprising: a processor configured to: receive an input specifying a schematic of user interface components of an application program (Abstract), see “a computer-implemented method, system and computer program product for generating a user interface. A sketch (e.g., wireframe) of a portion of a user interface is received”.
use a first group of one or more machine learning models to automatically identify the user interface components and associated properties specified in the input (p. 1, [0007]), see “the method additionally comprises generating and displaying a first set of intended final sketch renderings of the user interface using the first set of predicted intended sketches of the user interface based on historical data or a model trained to extract visual characteristics from existing user interface screens”.
based on the identified user interface components and the associated properties, use a second group of one or more machine learning models to automatically generate program code implementing the application program including the user interface components (p. 5, [0050]), see “in one embodiment, the model of the present invention is an encoder/decoder model that is trained on information from user interface metadata and screenshots of the user interfaces to translate the screenshots into a domain specific language and then into code”.
a memory coupled to the processor and configured to provide the processor with instructions (Abstract), see “a computer-implemented method, system and computer program product for generating a user interface. A sketch (e.g., wireframe) of a portion of a user interface is received”.
In regards to claim 23, Liu teaches:
A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for: receiving an input specifying a schematic of user interface components of an application program (Abstract), see “a computer-implemented method, system and computer program product for generating a user interface. A sketch (e.g., wireframe) of a portion of a user interface is received”.
using a first group of one or more machine learning models to automatically identify the user interface components and associated properties specified in the input (p. 1, [0007]), see “the method additionally comprises generating and displaying a first set of intended final sketch renderings of the user interface using the first set of predicted intended sketches of the user interface based on historical data or a model trained to extract visual characteristics from existing user interface screens”.
based on the identified user interface components and the associated properties, using a second group of one or more machine learning models to automatically generate program code implementing the application program including the user interface components (p. 5, [0050]), see “in one embodiment, the model of the present invention is an encoder/decoder model that is trained on information from user interface metadata and screenshots of the user interfaces to translate the screenshots into a domain specific language and then into code”.
Claim Rejections - 35 USC § 103
9. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
10. Claims 3, 6 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Liu in view of Nguyen et al., US 20160034441 (hereinafter Nguyen).
In regards to claims 1, 4 and 8 the rejections above are incorporated accordingly.
In regards to claim 3, Liu doesn't explicitly teach:
the one or more machine learning models of the first group remove noise from the input.
However, Nguyen teaches such use: (p. 6, [0074]), see “image dilation is a method of morphological image processing. Morphological image processing is a collection of non-linear operations related to the shape or morphology of features in an image. In general, a goal of morphological image processing is to remove imperfections in a digital image by accounting for the form and structure of the image. Morphological image processing techniques may utilize a structuring element that is a small binary image defined by a small matrix of pixels each with a value of zero or one”.
Liu and Nguyen are analogous art because they are from the same field of endeavor, user interface design.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the teaching of Liu and Nguyen before him or her, to modify the system of Liu to include the teachings of Nguyen, as a system for generating a user interface, and accordingly it would enhance the system of Liu, which is focused on Ai designed user interface, because that would provide Liu with the ability to provide improved systems, apparatuses, and methods for generating a user interface as suggested by Nguyen (p. 6, [0074], p. 12, [0153]).
In regards to claim 6, Liu teaches:
the one or more machine learning models of the first group automatically identify the user interface components and associated properties specified in the input at least in part by: loading the input using a computer vision library (p. 2, [0023]), see “the sketch is analyzed to predict a set of intended sketches using artificial intelligence based on historical data consisting of renderings of user interface designs and/or the user's asset library consisting of the user's previously designed user interfaces”.
Liu doesn't explicitly teach:
converting the input into a grayscale version of the input; and applying an edge detection algorithm to identity corresponding edges of the one or more shapes and their contours.
However, Nguyen teaches such use: (p. 4, [0051]), see “the next area of background pertains to edge detection. Edge detection refers to image processing techniques that detect edges in images. They may accomplish this by marking or identifying the points in a digital image at which the luminous intensity changes sharply. These points would indicate the presence of an edge. In the context of this disclosure, edges of interest may be, for example, edges of a UI button or edges of an icon image”.
Liu and Nguyen are analogous art because they are from the same field of endeavor, user interface design.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the teaching of Liu and Nguyen before him or her, to modify the system of Liu to include the teachings of Nguyen, as a system for generating a user interface, and accordingly it would enhance the system of Liu, which is focused on Ai designed user interface, because that would provide Liu with the ability to provide improved systems, apparatuses, and methods for generating a user interface as suggested by Nguyen (p. 6, [0074], p. 12, [0153]).
In regards to claim 10, Liu doesn't explicitly teach:
the identified shape is a repetitive shape.
However, Nguyen teaches such use: (p. 9, [0103]), see “Block 182 represents a process (discussed below with reference to FIG. 9) to identify repeated items that may represent list view elements” and (p. 11, [00124]), see “the first method may further include: identifying repeated elements in the graphical representation of the initial UI”.
Liu and Nguyen are analogous art because they are from the same field of endeavor, user interface design.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the teaching of Liu and Nguyen before him or her, to modify the system of Liu to include the teachings of Nguyen, as a system for generating a user interface, and accordingly it would enhance the system of Liu, which is focused on Ai designed user interface, because that would provide Liu with the ability to provide improved systems, apparatuses, and methods for generating a user interface as suggested by Nguyen (p. 6, [0074], p. 12, [0153]).
11. Claims 9, 13, 14, 17-19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Liu in view of Turek et al., US 2019/0317739 (hereinafter Turek).
In regards to claims 1, 4, 8, 15 and 16, the rejections above are incorporated accordingly.
In regards to claim 9, Liu doesn't explicitly teach:
the identified shape is filtered based on its dimensions being below a threshold.
However, Turek teaches such use: (p. 1, [0015], see once the basic requirements for the design are specified by the sketches and associated textual descriptions 104, the sketches and textual descriptions 104 are converted to one or more wireframes 106. Wireframes 106 provide a relatively low-fidelity, skeletal outline or framework for a GUI that are generating using a computer. Often, the wireframes 106 are limited to rudimentary shapes and/or lines representative of the size and position of visual components of the interface to be created. For example, rather than including an image, a wireframe may include a box with the label “image” to indicate the size and placement of the image” (emphasis added).
Liu and Turek are analogous art because they are from the same field of endeavor, user interface design.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the teaching of Liu and Turek before him or her, to modify the system of Liu to include the teachings of Turek, as a system for automatically generating user interface code, and accordingly it would enhance the system of Liu, which is focused on Ai designed user interface, because that would provide Liu with the ability to provide a GUI to users to facilitate feedback on its usability and appeal as suggested by Turek (p. 1, [0015], p. 15, [0146]).
In regards to claim 13, Liu doesn't explicitly teach:
the one or more machine learning models of the first group automatically identify the user interface components and associated properties specified in the input at least in part by extracting textual information from the one or more identified shapes.
However, Turek teaches such use: (Abstract), see “Methods and apparatus to automatically generate code for graphical user interfaces are disclosed. An example apparatus includes a textual description analyzer to encode a user-provided textual description of a GUI design using a first neural network”.
Liu and Turek are analogous art because they are from the same field of endeavor, user interface design.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the teaching of Liu and Turek before him or her, to modify the system of Liu to include the teachings of Turek, as a system for automatically generating user interface code, and accordingly it would enhance the system of Liu, which is focused on Ai designed user interface, because that would provide Liu with the ability to provide a GUI to users to facilitate feedback on its usability and appeal as suggested by Turek (p. 1, [0015], p. 15, [0146]).
In regards to claim 14, Liu doesn't explicitly teach:
the extracted textual information is utilized by the one or more machine learning models of the second group to determine a corresponding function associated with the one or more identified shapes.
However, Turek teaches such use: (Abstract), see “the example apparatus further includes a DSL statement generator to generate a DSL statement with a second neural network. The DSL statement is to define a visual element of the GUI design. The DSL statement is generated based on at least one of the encoded textual description or a user-provided image representative of the GUI design”.
Liu and Turek are analogous art because they are from the same field of endeavor, user interface design.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the teaching of Liu and Turek before him or her, to modify the system of Liu to include the teachings of Turek, as a system for automatically generating user interface code, and accordingly it would enhance the system of Liu, which is focused on Ai designed user interface, because that would provide Liu with the ability to provide a GUI to users to facilitate feedback on its usability and appeal as suggested by Turek (p. 1, [0015], p. 15, [0146]).
In regards to claim 17, Liu doesn't explicitly teach:
using the one or more machine learning models of the second group at least in part includes generating a corresponding token for the one or more identified shapes based on an output of the one or more machine learning models of the first group and the two-dimensional grid structure.
However, Turek teaches such use: (Abstract), see “the example apparatus further includes a DSL statement generator to generate a DSL statement with a second neural network. The DSL statement is to define a visual element of the GUI design. The DSL statement is generated based on at least one of the encoded textual description or a user-provided image representative of the GUI design”.
Liu and Turek are analogous art because they are from the same field of endeavor, user interface design.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the teaching of Liu and Turek before him or her, to modify the system of Liu to include the teachings of Turek, as a system for automatically generating user interface code, and accordingly it would enhance the system of Liu, which is focused on Ai designed user interface, because that would provide Liu with the ability to provide a GUI to users to facilitate feedback on its usability and appeal as suggested by Turek (p. 1, [0015], p. 15, [0146]).
In regards to claim 18, Liu doesn't explicitly teach:
the corresponding generated token for the one or more identified shapes is specified to a particular domain structured language.
However, Turek teaches such use: (p. 3, [0024], see “the DSL generation stage 204 analyzes and processes the textual and visual inputs 202 using a combination of AI models described further below to output a DSL (domain specific language) instructions 212 (e.g., executable code or script) (instructions which may be executed by a machine”.
Liu and Turek are analogous art because they are from the same field of endeavor, user interface design.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the teaching of Liu and Turek before him or her, to modify the system of Liu to include the teachings of Turek, as a system for automatically generating user interface code, and accordingly it would enhance the system of Liu, which is focused on Ai designed user interface, because that would provide Liu with the ability to provide a GUI to users to facilitate feedback on its usability and appeal as suggested by Turek (p. 1, [0015], p. 15, [0146]).
In regards to claim 19, Liu doesn't explicitly teach:
the program code is generated by a compiler based on the corresponding generated token for the one or more identified shapes.
However, Turek teaches such use: (p. 3, [0029], see “accordingly, as represented in the illustrated example of FIG. 2, a compilation and/or interpretation stage 222 involves running a compiler and/or interpreter to generate a full prototype of the final GUI 224. A compiler converts the programming language code 220 into binary code that can be run after compilation”.
Liu and Turek are analogous art because they are from the same field of endeavor, user interface design.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the teaching of Liu and Turek before him or her, to modify the system of Liu to include the teachings of Turek, as a system for automatically generating user interface code, and accordingly it would enhance the system of Liu, which is focused on Ai designed user interface, because that would provide Liu with the ability to provide a GUI to users to facilitate feedback on its usability and appeal as suggested by Turek (p. 1, [0015], p. 15, [0146]).
In regards to claim 20, Liu doesn't explicitly teach:
generating the application program that includes the user interface components based on the automatically generated program code.
However, Turek teaches such use: (p. 3, [0029], see “accordingly, as represented in the illustrated example of FIG. 2, a compilation and/or interpretation stage 222 involves running a compiler and/or interpreter to generate a full prototype of the final GUI 224. A compiler converts the programming language code 220 into binary code that can be run after compilation”.
Liu and Turek are analogous art because they are from the same field of endeavor, user interface design.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the teaching of Liu and Turek before him or her, to modify the system of Liu to include the teachings of Turek, as a system for automatically generating user interface code, and accordingly it would enhance the system of Liu, which is focused on Ai designed user interface, because that would provide Liu with the ability to provide a GUI to users to facilitate feedback on its usability and appeal as suggested by Turek (p. 1, [0015], p. 15, [0146]).
Allowable Subject Matter
12. Claim 11 and 12 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all the limitation of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
As per claim 11, prior art of record does not each and/or fairly suggest that “the identified shape is determined to be the repetitive shape for having a center coordinate difference with another identified shape that is below a set threshold”. The art of record does not expressly disclose such features.
As per claim 12, prior art of record does not each and/or fairly suggest that “the one or more machine learning models of the first group automatically identify the user interface components and associated properties specified in the input at least in part by redefining an identified shape of the one or more shapes by representing the identified shape by its center coordinates and dimensions instead of boundary coordinates associated with the identified shape”. The art of record does not expressly disclose such features.
Conclusion
13. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US Patent Application Publications
Sun 20220261226 teaches a method for generating web codes for a user interface (UI) based on a generative adversarial network (GAN) and a convolutional neural network (CNN). The method includes steps described below. A mapping relationship between display effects of a HyperText Markup Language (HTML) element and source codes of the HTML element is constructed. A location of an HTML element in an image I is recognized. Complete HTML codes of the image I are generated. The similarity between manually-written HTML codes and the generated complete HTML codes and the similarity between the image I and an image I1 generated by the generated complete HTML codes are obtained.
Kumar 20190250891 teaches automating GUI development from a GUI screen image that includes text information and one or more graphic user interface components. The GUI screen image is analyzed to extract text information and to identify the UI components included in the GUI screen. One or more text regions in the GUI screen image are detected and are replaced with placeholders. Images of one or more graphic user interface components in the GUI screen are extracted from the GUI screen image and are classified using a machine learning-based classifier.
14. Any inquiry concerning this communication or earlier communications from the examiner should be directed to Evral Bodden whose telephone number is 571-272-3455. The examiner can normally be reached on Monday to Friday from 9am to 5pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chat Do, can be reached at telephone number 571-272-3721. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automatedinterview-request-air-form.
If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EVRAL E BODDEN/Primary Examiner, Art Unit 2193
Prosecution Insights