DETAILED ACTION
This action is responsive to the application filed 9/6/2024.
Claims 1-24 are pending.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 7, 8, 19 and 20 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which a joint inventor regards as the invention.
Claims 7 and 19 recite that a graphical user interface component is one of a plurality of basic components, and Claims 8 and 20 recite that at least one of the components is a complex interface component. These claims are indefinite as the claims do not what constitutes a “basic” or “complex” interface component. Without any description of what is meant by “basic” and “complex,” it is impossible to determine the metes and bounds of the invention, as these are qualitative descriptors and therefore open to interpretation. Appropriate limitations defining what is meant by “basic” and “complex” is required in order for these claims to comply with the requirements of 35 U.S.C. § 112(b).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-4, 6-16 and 18-24 are rejected under 35 U.S.C. 103 as being unpatentable over Turek, et al., U.S. PGPUB No. 2019/0317739 (“Turek”), in view of Fuchs, U.S. PGPUB No. 2020/0174760 (“Fuchs”), and in view of Donohoe, et al., U.S. PGPUB No. 2019/0369969 (“Donohoe”).
With regard to Claim 1, Turek, in view of Fuchs teaches a computer-implemented method comprising:
obtaining, by one or more processors, an image depicting a graphical user interface component (Turek describes at [0023] that a visual input of an image can be provided to a GUI generation system, where [0021] describes that the images are of the GUI design);
determining, by the one or more processors and using a machine learning model, whether the graphical user interface component depicted in the obtained image can be implemented by any existing graphical user interface component stored in an asset database (Turek shows at Fig. 2 that the visual input is used in a process that generates a GUI, where [0024] describes the use of AI models. Fuchs teaches at [0018] that a code synthesis system can generate code for requested software components and identify existing software components for use in the identified task. [0025] describes that when no existing components are deemed sufficient for the required task, a new function will need to be created);
when determining that the graphical user interface component can be implemented by at least one existing graphical user interface component, retrieving from the asset database, by the one or more processors, auxiliary data associated with the at least one existing graphical user interface component ([0019] describes that the system can identify candidate components that can be used for the new component, along with test cases for the identified components);
when determining that the graphical user interface component cannot be implemented by any existing graphical user interface component: generating, by the one or more processors, a new graphical user interface component at least in part by generating an image associated with the new graphical user interface component and auxiliary data associated with the new graphical user interface component ([[0029] describes that the output of the system is the programming language code for the GUI, which can be compiled for execution. ; and
storing, by the one or more processors, the new graphical user interface component in the asset database ([0021] describes that a new function to be created is not yet a part of the component repository, indicating that once created, the new function is added to the repository of reusable components).
It would have been obvious to one of ordinary skill in the art at the time this application was filed to combine Fuchs with Turek. One of skill in the art would have sought the combination, to improve system functioning by enabling code reuse where applicable, which reduces the time and computing power needed to generate code for interface elements.
Turek does not teach generating, by the one or more processors, an abstract syntax tree based at least in part on either i) the auxiliary data associated with the at least one existing graphical user interface component; or ii) the auxiliary data associated with the new graphical user interface component. Donohoe teaches at [0093]-[0094] that a development system which compiles source code for a UI under development can generate an abstract syntax tree therefor.
It would have been obvious to one of ordinary skill in the art at the time this application was filed to combine Donohoe with Turek and Fuchs. Donohoe explains at [0093] that the use of an abstract syntax tree enables the system to more quickly provide updated previews for components under development. One of skill in the art would have sought to combine Donohoe with Turek and Fuchs, to improve user experience by including abstract syntax tree creation, in order to more quickly provide updated previews in cases where users make changes to generated components.
Claim 13 recites a system comprising memory and one or more processors communicatively coupled to the memory (Turek, Fig. 10) configured to carry out the method of Claim 1, and is similarly rejected.
With regard to Claim 2, Fuchs, in view of Donohoe teaches generating, by the one or more processors, and based on the abstract syntax tree, a code segment associated with the image depicting the graphical user interface component; and storing, by the one or more processors, the generated code segment in the asset database. Fuchs at [0021] describes that a new function to be created is not yet a part of the component repository, indicating that once created, the new function is added to the repository of reusable components. Donohoe teaches at [0093] that users can make edits to components, which are previewed using an abstract syntax tree generated from the component code.
It would have been obvious to one of ordinary skill in the art at the time this application was filed to combine Donohoe and Fuchs with Turek. One of skill in the art would have sought the combination, to improve the system both by including the computational efficiency of code reuse, and the quick preview updates enabled by the abstract syntax tree.
Claim 14 recites a system comprising memory and one or more processors communicatively coupled to the memory (Turek, Fig. 10) configured to carry out the method of Claim 2, and is similarly rejected.
With regard to Claim 3, Turek suggests that generating the code segment based on the abstract syntax tree includes using an attention layer trained based on training data stored in the asset database. [0039] describes an RNN layer which outputs fixed length, multidimensional vectors, where [0043] describes that the encoded representation is based on a suitable encoding method. [0032]-[0033] describe training the AI models.
Claim 15 recites a system comprising memory and one or more processors communicatively coupled to the memory (Turek, Fig. 10) configured to carry out the method of Claim 3, and is similarly rejected.
With regard to Claim 4, Turek suggests that the machine learning model is a convolutional neural network including classification layers and feature generating layers. [0037] describes that the image analysis is carried out by convolutional neural networks, where one of skill in the art would understand that such an architecture generally includes classification and feature generating layers, and [0045] describes the CNN output as a multidimensional tensor of feature maps.
Claim 16 recites a system comprising memory and one or more processors communicatively coupled to the memory (Turek, Fig. 10) configured to carry out the method of Claim 4, and is similarly rejected.
With regard to Claim 6, Turek teaches that the feature generating layers are configured to generate the auxiliary data associated with the new graphical user interface component, and the auxiliary data associated with the new graphical user interface component is indicative of a plurality of constituent graphical user interface components. [0029] describes the output of the source code as well as compiled binary code. As shown in Fig. 2, a final GUI recognized from an image is a screen incorporating several individual elements, such as a play/pause control and a video navigation bar.
Claim 18 recites a system comprising memory and one or more processors communicatively coupled to the memory (Turek, Fig. 10) configured to carry out the method of Claim 6, and is similarly rejected.
With regard to Claim 7, Turek teaches that each of the plurality of constituent graphical user interface components is one of a plurality of basic graphical user interface components. Fig. 2 shows a final GUI recognized from an image is a screen incorporating several individual elements, such as a play/pause control and a video navigation bar, which are basic elements of a video playing GUI as understood by those of ordinary skill in the art.
Claim 19 recites a system comprising memory and one or more processors communicatively coupled to the memory (Turek, Fig. 10) configured to carry out the method of Claim 7, and is similarly rejected.
With regard to Claim 8, Fuchs teaches that at least one of the plurality of constituent graphical user interface components is one of a plurality of existing complex graphical user interface components stored in the asset database. –[0022] describes that components can be complex enough to receive multiple types of input to provide defined output.
It would have been obvious to one of ordinary skill in the art at the time this application was filed to combine Fuchs with Turek. One of skill in the art would have sought the combination, to improve system functioning by enabling code reuse where applicable, which reduces the time and computing power needed to generate code for interface elements.
Claim 20 recites a system comprising memory and one or more processors communicatively coupled to the memory (Turek, Fig. 10) configured to carry out the method of Claim 8, and is similarly rejected.
With regard to Claim 9, Turek teaches that obtaining, by the one or more processors, the image depicting the graphical user interface component, includes identifying the image depicting the graphical user interface component within a larger image depicting a plurality of graphical user interface components. [0017] describes that the final GUI which is generated includes all animations and functionality. Fig. 2 shows an example interface of a video playback GUI, which includes individual components such as a play button and navigation controls.
Claim 21 recites a system comprising memory and one or more processors communicatively coupled to the memory (Turek, Fig. 10) configured to carry out the method of Claim 9, and is similarly rejected.
With regard to Claim 10, Fuchs teaches that determining whether the graphical user interface component depicted in the obtained image can be implemented by any of the existing graphical user interface components includes: computing a plurality of metrics each indicative of a quality of match associated with a respective one of a plurality of existing complex graphical user interface components, selecting a maximum metric from the plurality of metrics, and comparing the maximum metric to a threshold. [0051] describes that candidate components can be identified as components having a predetermined degree of similarity. [0053] describes that from among the similar components, a component can be selected for adaptation when the existing component is determined to have a sufficiently similar signature to the new function.
It would have been obvious to one of ordinary skill in the art at the time this application was filed to combine Fuchs with Turek and Donohoe. One of skill in the art would have sought the combination, to improve system functioning by enabling code reuse where applicable, which reduces the time and computing power needed to generate code for interface elements.
Claim 22 recites a system comprising memory and one or more processors communicatively coupled to the memory (Turek, Fig. 10) configured to carry out the method of Claim 10, and is similarly rejected.
With regard to Claim 11, Turek suggests that the machine learning model is trained using a dataset stored in the asset database. [0069] describes that in some examples, AI models come pre-trained, suggesting that in other embodiments, AI models can be locally trained, using data stored in a local data store which also stores programming components.
Claim 23 recites a system comprising memory and one or more processors communicatively coupled to the memory (Turek, Fig. 10) configured to carry out the method of Claim 11, and is similarly rejected.
With regard to Claim 12, Turek teaches that the dataset includes the image depicting the graphical user interface component. Fig. 6 shows that image analysis is carried out within the computing device, thereby storing the image in the local data store.
Claim 24 recites a system comprising memory and one or more processors communicatively coupled to the memory (Turek, Fig. 10) configured to carry out the method of Claim 12, and is similarly rejected.
Claims 5 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Turek, in view of Fuchs, in view of Donohoe, and in view of Azimi, et al., U.S. PGPUB No. 2023/0131183 (“Azimi”).
With regard to Claim 5, Azimi teaches that the classification layers are configured to classify the graphical user interface component as (i) one of a plurality of existing complex graphical user interface components stored in the asset database or (ii) one of a plurality of basic graphical user interface components. Fig. 4 shows that UI interfaces can be classified into screens, which have categories and attributes, where screens are further processed to also classify the components which make them up. [0055] similarly describes this process, and [0053] describes that the process uses AI tools for classification and decomposition.
It would have been obvious to one of ordinary skill in the art at the time this application was filed to combine Azimi with Turek, Fuchs and Donohoe. The background of Azimi describes that the processes therein aim to improve upon the prior art methods of evaluating UIs and how changes thereto will impact end users. One of skill in the art would have therefore sought to combine Azimi with Turek, Fuchs and Donohoe, to improve system functioning by offering developers improved UI testing and analysis.
Claim 17 recites a system comprising memory and one or more processors communicatively coupled to the memory (Turek, Fig. 10) configured to carry out the method of Claim 5, and is similarly rejected.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KEITH D BLOOMQUIST whose telephone number is (571)270-7718. The examiner can normally be reached M-F, 8:30-5 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kieu Vu can be reached at 571-272-4057. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KEITH D BLOOMQUIST/Primary Examiner, Art Unit 2171
6/4/2026
Prosecution Insights