DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This Office Action is responsive to the communications filed on 22 July 2024. Claims 1-20 are pending.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Independent claim 1 recites “determining whether a similarity threshold is reached by comparing a candidate digital image generated using the candidate markup code with the digital image.”
The limitations of “determining whether a similarity threshold is reached by comparing a candidate digital image generated using the candidate markup code with the digital image”, as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, other than reciting “by a processing device,” nothing in the claim element precludes the step from practically being performed in the mind. For example, “comparing images” and “determining similarity” in the context of this claim encompass a user mentally making a mental judgement about images viewed by the user. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recited an abstract idea.
This judicial exception is not integrated into a practical application. In particular, the claim recites additional elements – using a processing device to perform the extracting, generating, and outputting steps. The processing device in these steps is recited at a high-level of generality (i.e., as a generic processor performing a generic computer functions of data analysis, classification, data organization, and receiving results) such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Independent claim 1 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using a processing device to perform the additional steps amount to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Therefore, independent claim 1 is not patent eligible.
Independent claim 11
Independent claim 11 recites “determining a similarity threshold is reached by comparing a missing candidate digital image generated using the missing candidate markup code with the digital image.”
The limitations of “determining a similarity threshold is reached by comparing a missing candidate digital image generated using the missing candidate markup code with the digital image”, as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, other than reciting “a processing device,” nothing in the claim element precludes the step from practically being performed in the mind. For example, “comparing images” and “determining similarity” in the context of this claim encompass a user mentally making a mental judgement about images viewed by the user. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recited an abstract idea.
This judicial exception is not integrated into a practical application. In particular, the claim recites additional elements – using a processing device to perform the extracting, generating, identifying, initiating, and outputting steps. The processing device in these steps is recited at a high-level of generality (i.e., as a generic processor performing a generic computer functions of data analysis, classification, data organization, and receiving results) such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Independent claim 11 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using a processing device to perform the additional steps amount to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Therefore, independent claim 1 is not patent eligible.
Independent claim 11 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using a processing device to perform the additional steps amount to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Therefore, independent claim 1 is not patent eligible.
Independent claim 14
Independent claim 14 recites “generating a layout extraction prompt to instruct one or more machine-learning models to extract layout data based on elements included in a digital image, the layout data describing bounding boxes of the elements, elements classes of the elements, and a hierarchical layout structure of the elements.”
The limitations of “generating a layout extraction prompt to instruct one or more machine-learning models to extract layout data based on elements included in a digital image, the layout data describing bounding boxes of the elements, elements classes of the elements, and a hierarchical layout structure of the elements”, as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, other than reciting “by processing device,” nothing in the claim element precludes the step from practically being performed in the mind. For example, “generating a layout-extraction prompt” in the context of this claim encompass a user mentally making a mental judgement about what elements are to be extracted from an image. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recited an abstract idea.
This judicial exception is not integrated into a practical application. In particular, the claim recites additional elements – using a processing device to perform the receiving and generating steps. The processing device in these steps is recited at a high-level of generality (i.e., as a generic processor performing a generic computer functions of data analysis, classification, data organization, and receiving results) such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Independent claim 14 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using a processing device to perform the additional steps amount to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Therefore, independent claim 14 is not patent eligible.
Dependent claims 2-10, 12-13 and 15-20 do not include elements that amount to significantly more than the abstract idea and are also rejected under the same rational.
Claims 14-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
In summary, Claim 14-20 recite a “computer-readable storage media” storing instructions that perform various functions. In the Specification of the present application, the “computer readable medium” is expressly defined as including transmission media (see Page 33, Paragraph 0081 first sentence). Thus, the broadest, reasonable interpretation of “computer readable medium” encompasses nonstatutory subject matter (transmission media) that is unpatentable under 35 U.S.C. 101.
Accordingly, Claim 14-20 fail to recite statutory subject matter under 35 U.S.C. 101.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Kumar et al. (Hereinafter, Kumar, US 2019/0250891 A1).
Per claim 1, Kumar discloses a method (paragraph [0005], “The present disclosure relates to application development, and more particularly, to techniques for automating the development of a graphic user interface (GUI) for an application from design documents, such as one or more images or sketches for one or more GUI screens of the application. Various inventive embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, code, or instructions executable by one or more processors, and the like.”) comprising:
extracting (e.g., steps 604-614 as shown in Fig. 6; paragraphs [0137-0142]), by a processing device(e.g., processing subsystem 1904 as shown in Fig. 19; paragraph [0200]), layout data from a digital image(e.g., step 618 as shown in Fig. 6; paragraph [0144], “ At 618, a GUI model may be generated for the GUI based upon the text content items and corresponding locations, the classified UI components and corresponding locations, and the layout for the GUI screen ... “; Kumar determines at least the location of text and UI components.), the layout data describing a layout of elements included in the digital image (Abstract, “ … The GUI screen image is analyzed to extract text information and to identify the UI components included in the GUI screen. One or more text regions in the GUI screen image are detected and are replaced with placeholders. Images of one or more graphic user interface components in the GUI screen are extracted from the GUI screen image and are classified using a machine learning-based classifier ... “; paragraph [0142]; paragraph [0144], “…The information stored in the GUI model can be used by a downstream consumer to generate an implementation of the GUI. In some implementations, the GUI model may be generated as metadata for a GUI. In some implementations, the GUI model may be described in a data-interchange format that is language independent, such as JavaScript Object Notation (JSON) format. “; paragraph [0154]; Examiner’s Note: Kumar disclose extracting attributes of text content items and UI components, such as the locations, sizes, types, functions, and the like );
generating, by a processing device, markup language code (paragraph [0016]; paragraph [0055], “… The GUI model (e.g., in JSON format) can also be used to generate code in different programming languages, such as markup languages (e.g., HTML or XML) or stylesheet languages (e.g., cascading style sheet (CSS)”). “; paragraph [0075]) by generating one or more iterations of candidate markup code using a machine-learning model based on the digital image and the layout data (e.g., step 620 as shown in Fig. 6; paragraph [0098], “…The classification results may be compared with the annotations associated with the training sample. If the classification results do not match the annotations, feedback may be provided to the classification layer and/or the feature extraction layers (e.g., the convolution layers) to adjust the parameters of the deep CNN using, for example, the back propagation technique. The above-described training process may be repeated for each training sample ... “; paragraph [0142], “… As described above, a clustering module (e.g., clustering module 356) may perform the grouping recursively in a bottom up manner based upon certain pre-defined rules ... “; paragraph [0145], “ At 620, source code for implementing the GUI may be automatically generated based upon the GUI model. In some embodiments, the source code for implementing the GUI may be generated based on certain code generation templates… “;Examiner’s Note: Kumar teaches calculating distance metrics and similarity/dissimilarity thresholds as shown in step 512 as shown in Fig. 5; paragraph [0108-0109]; paragraph [0128]);
determining whether a similarity threshold is reached by comparing a candidate digital image generated using the candidate markup code with the digital image(paragraph [0025]; paragraphs [0108-0109]; paragraph [0131];paragraph [0152];); and
outputting, by the processing device, the markup language code responsive to determining the similarity threshold is reached(paragraph [0082]).
Per claim 2, Kumar discloses the method as described in claim 1, wherein the layout data defines bounding boxes of the elements (Kumar, paragraph [0121]; paragraph [0137]; paragraph [0149]; paragraph [0153]), elements classes of the elements (Kumar, paragraph [0082]; paragraph [0092]; paragraph [0126-0127]), and a hierarchical layout structure of the elements (Kumar, paragraph [0012], “… For each GUI screen, the model may also include information about the structure of the GUI screen, such as information identifying a hierarchical organization of the user interface components and text content items on the GUI screen ... “; paragraph [0025]; paragraph [0068]).
Per claim 3, Kumar discloses the method as described in claim 1, wherein the extracting includes generating a layout extraction prompt configured to initiate a machine-learning model to generate at least a portion of the layout data using the digital image(Kumar, paragraph [0157]) .
Per claim 4, Kumar discloses the method as described in claim 3, wherein the layout extraction prompt includes instructions to cause the machine-learning model to:
identify distinct sections of the digital image (Kumar, paragraph [007];paragraph [0097], “To extract features from an input image (e.g., a training image or a GUI screen image), a convolution layer may first be used to convolute the input image with one or more filters to generate one or more feature maps for an input image, such as detecting edges or gradients in an image …”; paragraph [0087] ; Examiner’s Note: Header and body sections are identified in an email. );
determine relative position of the elements using spatial descriptors (Kumar, paragraph [0011], “… In some embodiments, the text content item in the portion of the image may be replaced with a pixel array having a pre-defined pattern of pixels or having pixel values of the background of the portion of the image. This pixel array replacement not only causes the actual text content to be removed from the image portion but also identifies the location and boundaries of the text content within the image… “; paragraph [0022]; paragraph [0069]);
identify text alignment and formatting attributes(Kumar, paragraph[0097]; paragraph [0122]; paragraph [0139]);
recognize and describe lines, borders, dividers, or shapes (Kumar, paragraph [0012]); or
explicitly specify a respective side, with respect to which, elements are located within the digital image(Kumar, paragraph [0068]).
Per claim 5, Kumar discloses the method as described in claim 1, wherein the generating includes generating a markup language prompt configured to initiate the machine-learning model to generate the candidate markup code (Kumar, paragraph [0079], “ For example, code generator 126 may be configured to receive one or more files comprising markup code corresponding to GUI model 124 and output a GUI implementation 110 comprising one or more source code files by translating the markup code (e.g., XML) into (high-level) source code (e.g., Java, C++, or other programming language). “).
Per claim 6, Kumar discloses the method as described in claim 5, wherein the markup language prompt is configured to instruct the machine-learning model to use the layout data as a guiding framework during generation of the markup language code.
Per claim 7, Kumar discloses the method as described in claim 5, wherein the markup language prompt is configured to instruct the machine-learning model to provide the markup language code as a comprehensive output of the digital image (Kumar, paragraph [0016], “… For example, a first consumer may use the GUI model for automatically generating an executable for a first platform (e.g., iOS®) and a second consumer may use the same GUI model to automatically generate a second executable for a different platform (e.g., Android®). The GUI model (e.g., in JSON format) can also be used to generate code in different programming languages, such as markup languages (e.g., HTML or XML) or stylesheet languages (e.g., cascading style sheet (CSS)). “; paragraph [0055]).
Per claim 8, Kumar discloses the method as described in claim 5, wherein the markup language prompt is configured to instruct the machine-learning model to guide inclusion of at least one placeholder having dimensions based on those of a respective object in the digital image (Kumar, Abstract, “… One or more text regions in the GUI screen image are detected and are replaced with placeholders …“; paragraph [0011]; paragraph [0017]; paragraph [0022] ).
Per claim 9, Kumar discloses the method as described in claim 5, wherein the markup language prompt is configured to instruct the machine-learning model to maintain spatial properties of the elements of the digital image (Kumar, paragraph [0121]; paragraph [0137], “… In some embodiments, a fully convolutional network model may be used to detect text regions in the GUI screen image and determine the locations (e.g., the coordinates of the bounding boxes) and/or the dimensions of the text regions. “; paragraph [0149]; paragraph [0151]; Kumar discloses determining maintaining the coordinates of bounding boxes.).
Per claim 10, Kumar discloses the method as described in claim 1, wherein the generating the one or more iterations of candidate markup code using the machine-learning model includes:
identifying a missing element based on the comparing of the candidate digital image generated using the candidate markup code with the digital image(Kumar, paragraph [0107]);
initiating generation of missing candidate markup code as part of the one or more iterations of generating the candidate markup code based on the missing element (Kumar, paragraph [0108]); and
the comparing includes comparing a respective said candidate markup image generated based on the missing candidate markup code with the digital image (Kumar, paragraph [0108], “ … The features extracted from the feedback image of the misclassified or undetected UI component may also be mapped to a data point in the feature space. Distances between the data point representing the features extracted from the image of the misclassified or undetected UI component and the cluster center of the set of clusters may be calculated to determine the similarity or dissimilarity between the misclassified or undetected UI component and the UI components already included in the training samples ... “).
Per claim 11, Kumar discloses a computing device comprising:
a processing device (e.g., processing subsystem 1904 as shown in Fig. 19; paragraph [0200]); and
a computer-readable storage medium storing instructions that, responsive to execution by the processing device (e.g., storage subsystem 1918 as shown in Fig. 19; paragraph [0198]), causes the processing device to perform operations including:
extracting (e.g., steps 604-614 as shown in Fig. 6; paragraphs [0137-0142]) layout data from a digital image(e.g., step 618 as shown in Fig. 6; paragraph [0144], “ At 618, a GUI model may be generated for the GUI based upon the text content items and corresponding locations, the classified UI components and corresponding locations, and the layout for the GUI screen ... “; Kumar determines at least the location of text and UI components.), the layout data describing a layout of elements included in the digital image (Abstract, “ … The GUI screen image is analyzed to extract text information and to identify the UI components included in the GUI screen. One or more text regions in the GUI screen image are detected and are replaced with placeholders. Images of one or more graphic user interface components in the GUI screen are extracted from the GUI screen image and are classified using a machine learning-based classifier ... “; paragraph [0142]; paragraph [0144], “…The information stored in the GUI model can be used by a downstream consumer to generate an implementation of the GUI. In some implementations, the GUI model may be generated as metadata for a GUI. In some implementations, the GUI model may be described in a data-interchange format that is language independent, such as JavaScript Object Notation (JSON) format. “; paragraph [0154]; Examiner’s Note: Kumar disclose extracting attributes of text content items and UI components, such as the locations, sizes, types, functions, and the like);
generating candidate markup code using one or more machine-learning models based on the digital image and the layout data (paragraph [0016]; paragraph [0055], “… The GUI model (e.g., in JSON format) can also be used to generate code in different programming languages, such as markup languages (e.g., HTML or XML) or stylesheet languages (e.g., cascading style sheet (CSS)”). “; paragraph [0075]);
identifying a missing element by comparing the digital image with a candidate digital image generated through execution of the candidate markup code (paragraph [0107], “In some cases, model generation system 330 may not accurately detect or classify the UI components in the GUI screen image. A feedback process may be used to provide feedback to model generation system 330 to correct the generated GUI model. For example, a developer may review GUI model 306 and identify undetected or misclassified UI components, and provide the image of the identified undetected or misclassified UI components and the correct labels to model generation system 330 through an optional feedback module 304 ...”; paragraph [0130] );
initiating generation of missing candidate markup code based on the missing element using the one or more machine-learning models(paragraph [0107], “… For example, if there is a UI component that has been misclassified or undetected by model generation system 330 (more specifically, object detection module 354), the image of the misclassified or undetected UI component or the GUI screen image that includes the misclassified or undetected UI component may be provided to model generation system 330 through REST service 340, along with the correct label for the misclassified or undetected UI component. The feedback information may be used by feature extraction engine 366 to extract features (e.g., feature maps or feature vectors) from the image of the misclassified or undetected UI component or the GUI screen image as described above. In some embodiments, the features extracted from the user feedback may be sent to a feature clustering module 372. “; ) ;
determining a similarity threshold is reached by comparing a missing candidate digital image generated using the missing candidate markup code with the digital image(paragraph [0108], “…Distances between the data point representing the features extracted from the image of the misclassified or undetected UI component and the cluster center of the set of clusters may be calculated to determine the similarity or dissimilarity between the misclassified or undetected UI component and the UI components already included in the training samples. If one of the distances is below a threshold value, the image of the misclassified or undetected UI component may be added to the training samples and the retraining of the machine learning-based model may be triggered …“; paragraph [0117]; paragraph [0131]); and
outputting markup language code based on the missing candidate markup code (paragraph [0082]).
Per claim 12, Kumar discloses the computing device as described in claim 11, wherein the extracting is performed using the one or more machine-learning models (paragraph [0070]; paragraph [0098]; paragraph [0115]).
Per claim 13, Kumar discloses the computing device as described in claim 11, wherein the digital image is a webpage or an email(e.g., GUI screen 200 as shown in Fig. 2; paragraph [0057]; paragraph [0087]).
Per claim 14, Kumar discloses one or more computer-readable storage media storing instructions(e.g., storage subsystem 1918 as shown in Fig. 19; paragraph [0198]) that, responsive to execution by a processing device (e.g., processing subsystem 1904 as shown in Fig. 19; paragraph [0200]), causes the processing device to perform operations comprising:
generating a layout extraction prompt to instruct one or more machine-learning models to extract layout data based on elements included in a digital image(e.g., step 502 as shown in Fig. 5; paragraph [0121]; paragraph [0137], “… In some embodiments, a fully convolutional network model may be used to detect text regions in the GUI screen image and determine the locations (e.g., the coordinates of the bounding boxes) and/or the dimensions of the text regions. “; paragraph [0149]; paragraph [0151]; paragraph [0119]), the layout data describing bounding boxes of the elements( paragraph [0121]; paragraph [0137]; paragraph [0149]; paragraph [0153]), elements classes of the elements(paragraph [0082]; paragraph [0092]; paragraph [0126-0127]), and a hierarchical layout structure of the elements (paragraph [0012], “… For each GUI screen, the model may also include information about the structure of the GUI screen, such as information identifying a hierarchical organization of the user interface components and text content items on the GUI screen ... “; paragraph [0025]; paragraph [0068]);
receiving the layout data from the one or more machine-learning models(paragraph [0059], “ As shown in FIG. 1, system 100 may include a model generation system (MGS) 102 that is configured to receive one or more GUI screen images 104 for a GUI as input and generate a GUI model 124 for the GUI based upon the one or more GUI screen images 104 … “);
generating a markup language prompt based on the layout data and the digital image, the markup language prompt configured to instruct the one or more machine-learning models to generate markup language code(paragraph [0159]); and
receiving the markup language code from the one or more machine-learning models(paragraph [0079], “… For example, code generator 126 may be configured to receive one or more files comprising markup code corresponding to GUI model 124 and output a GUI implementation 110 comprising one or more source code files by translating the markup code (e.g., XML) into (high-level) source code (e.g., Java, C++, or other programming language).“; paragraph [0148]).
Per claim 15, Kumar discloses the one or more computer-readable storage media as described in claim 14, wherein the layout extraction prompt includes instructions to cause the machine-learning model to:
identify distinct sections of the digital image; determine relative position of the elements using spatial descriptors(paragraph [007];paragraph [0097], “To extract features from an input image (e.g., a training image or a GUI screen image), a convolution layer may first be used to convolute the input image with one or more filters to generate one or more feature maps for an input image, such as detecting edges or gradients in an image …”; paragraph [0087] ; Examiner’s Note: Header and body sections are identified in an email.);
identify text alignment and formatting attributes; recognize and describe lines, borders, dividers, or shapes (paragraph [0011], “… In some embodiments, the text content item in the portion of the image may be replaced with a pixel array having a pre-defined pattern of pixels or having pixel values of the background of the portion of the image. This pixel array replacement not only causes the actual text content to be removed from the image portion but also identifies the location and boundaries of the text content within the image… “; paragraph [0022]; paragraph [0069]); or
explicitly specify a respective side, with respect to which, elements are located within the digital image(paragraph [0068]).
Per claim 16, Kumar discloses the one or more computer-readable storage media as described in claim 14, wherein the markup language prompt is configured to instruct the one or more machine-learning models to use the layout data as a guiding framework during generation of the markup language code(paragraph [0071]).
Per claim 17, Kumar discloses the one or more computer-readable storage media as described in claim 14, wherein the markup language prompt is configured to instruct the one or more machine-learning models to provide the markup language code as a comprehensive output of the digital image (paragraph [0016], “… For example, a first consumer may use the GUI model for automatically generating an executable for a first platform (e.g., iOS®) and a second consumer may use the same GUI model to automatically generate a second executable for a different platform (e.g., Android®). The GUI model (e.g., in JSON format) can also be used to generate code in different programming languages, such as markup languages (e.g., HTML or XML) or stylesheet languages (e.g., cascading style sheet (CSS)). “; paragraph [0055]).
Per claim 18, Kumar discloses the one or more computer-readable storage media as described in claim 14, wherein the markup language prompt is configured to instruct the one or more machine-learning models to guide inclusion of at least one placeholder having dimensions based on those of a respective object in the digital image(Abstract, “… One or more text regions in the GUI screen image are detected and are replaced with placeholders …“; paragraph [0011]; paragraph [0017]; paragraph [0022]).
Per claim 19, Kumar discloses the one or more computer-readable storage media as described in claim 14, wherein the markup language prompt is configured to instruct the one or more machine-learning models to maintain spatial properties of the elements of the digital image(Kumar, paragraph [0121]; paragraph [0137], “… In some embodiments, a fully convolutional network model may be used to detect text regions in the GUI screen image and determine the locations (e.g., the coordinates of the bounding boxes) and/or the dimensions of the text regions. “; paragraph [0149]; paragraph [0151]; Kumar discloses determining maintaining the coordinates of bounding boxes.).
Per claim 20, Kumar discloses the one or more computer-readable storage media as described in claim 14, further comprising generating the digital image for display in a user interface by executing the markup language code by one or more processing devices (paragraph [0010]; paragraph [0157]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Davit Soselia, Khalid Saifullah, and Tianyi Zhou. November 3, 2023. “Learning UI-to-Code Reverse Generator Using Visual Critic Without Rendering.” arXiv:2305.14637, https://arxiv.org/pdf/2305.14637, retrieved May 29, 2026.
Yuxuan Wan, Chaozheng Wang, Yi Dong, Wenxuan Wang, Shuqing Li, Yintong Huo, Michael R. Lyu. “Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approach.” ArXiv, https://arxiv.org/abs/2406.16386v1. Retrieved May 26, 2026.
Tony Beltramelli. “pix2code: Generating Code from a Graphical User Interface Screenshot”, September 19, 2017. arXiv:1705.07962, https://arxiv.org/abs/1705.07962, retrieved May 29, 2026.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DARRIN HOPE whose telephone number is (571)270-5079. The examiner can normally be reached Mon-Thr - 6:45-4:15, Fri - 6:45-3:15, Alt. Fri Off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen S Hong can be reached at (571)272-4124. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
DARRIN HOPE
Examiner
Art Unit 2178
/STEPHEN S HONG/Supervisory Patent Examiner, Art Unit 2178
Prosecution Insights