DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 3 and 13 have been canceled and as such all rejections to the canceled claims are withdrawn as moot.
Response to Arguments
Applicant’s arguments, see applicant’s correspondence, filed 3/12/2026, with respect to the rejection(s) of claim(s) 1 and 11 under 35 U.S.C. 103 based on Park, Bialota and Pardeshi have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Kim.
Applicant's arguments filed 3/12/2026 with respect to claims 6 and 20 have been fully considered but they are not persuasive.
Regarding claim 6, Applicant argues that Park modified by Bialota and Pardeshi fails to teach or render obvious that the features of generating a modified image using first and second predefined prompt, such that the first predefined prompt is reflected in a first region of the original image, and the second predefined prompt is reflected in a second region of the original image (See pages 10 and 17-19 of applicant’s correspondence filed 3/12/2026). Examiner respectfully disagrees.
In particular, Paradeshi discloses a predefined prompt comprises a first predefined prompt and a second predefined prompt, (Paradeshi, ¶56 discloses analyzing a GUI to determine a plurality of visual and functional features, such as structure, design, color set, images, tokens and text elements – i.e. “predefined prompts” prior to generative model) and wherein the one or more instructions, when executed by the one or more processors, further cause the electronic device to generate, by using the generative model, the modified image (Pardeshi, ¶56 discloses using the parameters to generate a second GUI – see Figs. 1-2 of Pardesh showing resulting translated GUIs from laid out elements of original GUI) such that the first predefined prompt is reflected in a first region of the original image, and the second predefined prompt is reflected in a second region of the original image (Pardeshi, ¶56: GUI analyzed to identify visual and functional features of the first GUI, including structure, design, color set, images, tokens and text elements, with these features transformed into feature vectors and encoded into a latent space for generation of second GUI, where context for elements or features of this first GUI can be determined 410 in order to determine relations of elements, such as may cause these elements to be kept together or able to be divided across multiple pages or screen of a generated interface – see further Fig. 1 showing input GUI 102, having elements in first and second regions, any number of which are converted to vector parameters for generating the second GUI). Examiner notes that the way the claim is drafted, the first and second regions relate to the original image, as opposed to the modified image. The claim merely states that the predefined prompts (i.e. vector parameters fed into the generative model of Pardeshi) are “reflected” in the regions of the original image (i.e. not the modified image). Accordingly, applicant’s arguments are not persuasive.
Regarding claim 20, Applicant argues that Park modified by Bialota and Pardeshi fails to teach or render obvious that the changed property information is obtained from the recited generative model (See pages 10 and 17-19 of applicant’s correspondence filed 3/12/2026). Examiner respectfully disagrees.
Applicant’s claim merely attaches a generative model to perform the image changes and editing in place of a pre-programmed algorithm. The editing of a number of elements of the GUI interface using GUI layout and design parameter inputs to a generative model which then generates the modifications in place of a particular defined algorithm, however, is well known. In particular, Pardeshi disclose generating a modified image corresponding to changed property information of the UI object, where the changed property information is obtained from the generative model. Pardeshi, ¶61 discloses that a GUI is analyzed to obtain a the visual characteristics of elements, including structure and design, color sets and text elements, and transforming those elements into latent space used by a generative network to generate a second GUI, which infers the appropriate layout of the second GUI – see e.g. Fig. 1 showing input GUI 102 to translated GUIs changing layout and arrangement of data (see Pardeshi, ¶56: In at least one embodiment, a screen divider 318 can take as input text and asset-related conditions, and can learn whether any of these assets, test, or images can, or should, be scaled down or adjusted as necessary for a design layout for a generated output GUI 316, and can determine how much real estate is needed to render a particular interface, which can then be used to determine how many screens or pages may be required for an interface for a target platform; ¶61 further discloses analyzing GUI to identify visual features, including structure and design, as well as text elements, color set, and tokens, and transforming the features into vectors encoded in latent space defining constraints for generation of a second GUI, which infers an image having appropriate layout for each page of the second GUI; Fig. 2 also showing text tokens recognized separately from images, where text shown overlaid on input GUI 202). In other words, Pardeshi discloses that the generative model itself generates the changes to the UI objects and images of the first GUI to provide the second GUI, as opposed to merely another type of algorithm. Accordingly, the combination of the teachings of using the generative model of Pardeshi with the use of a variety of input parameters to obtain a changed property information for a second GUI display of data with the teachings of Park and Bialota teaching the particulars of the arrangement and modifications of the image data. Accordingly, applicant’s arguments are not persuasive.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 4-6, 8, 11, 12, and 14-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over
Park et al. (US 2022/0058038 A1) in view of
Bialota (US 2015/0029206 A1) and in further view of
Pardeshi et al. (US 2021/0406673 A1) and
Kim et al. (US 2020/0265616 A1).
Regarding claim 11, Park discloses:
An electronic device for changing a background screen, (Park, Abstract and ¶9: configuring background screen of electronic device) the electronic device comprising:
A display (Park, Fig. 1 and ¶33: electronic device includes display module; ¶40: display module including display);
Memory storing one or more instructions (Park, Fig. 1 and ¶33: memory); and
One or more processors operatively coupled to the memory and the display, and configured to execute the one or more instructions stored in the memory, wherein the one or more instructions, when executed by the one or more processors, cause the electronic device to:
(Park, Fig. 1 and ¶33: processor and memory; ¶34: processor executes software; ¶36: memory stores software used by processor)
Obtain an original image designated as the background screen (Park, Fig. 7A and ¶¶93-95 discloses second object corresponding to dynamic wall paper having layered objects, including e.g. b-th object corresponding to image content and a-th corresponding to particle content, together an “original image”; ¶110: second object corresponds to live wallpaper screen);
Obtain(Park, ¶110: specific second data may be determined depending on the weather reflecting a user's location)
Obtain, based on (Park, ¶95: particle content for snowy weather; ¶110: specific second data may be determined depending on weather reflecting a user’s location, e.g. if weather reflecting user’s location is “snowy”, the second data may correspond to display of “snowy” as a particle);
obtain display area information of a user interface (UI) object that is overlaid on the original image (Park, Fig. 3 and ¶64: processor of electronic device identifies a first object and first data on the first object, where the first object may be an initial home screen of the electronic device, and may correspond to a screen including a plurality of layers, the first object may include three sheets of layers of the home screen and data (e.g., first data) of each layer, and the data of each layer is data about an application being displayed on each layer, and may include coordinates and attributes of widgets and icons (e.g., application name, application type, package information, and icon color; Fig. 5 and ¶84 discloses home screen composed of individual layers, where “the first data is data for the application being displayed on an individual layer, and may include coordinates of widgets and icons, and sizes or attributes of icons of the widgets and applications (e.g., application name, application type, package information, icon colors)”);
Generate a modified image corresponding to (Park, ¶¶95-96 discloses the processor analyzing the objects such that the synthesis of the a-th to f-th objects may be changed; Fig 10A and ¶110 discloses third object 1030 using weather of user’s location, with data of first object used as obstacle)
Wherein in the modified image, the predefined prompt is reflected in a display area where the UI object is displayed in accordance with a first degree, and the predefined prompt is reflected in a remaining area other than the display area in accordance with a second degree that is different from the first degree (Park, Fig 10A and ¶110 discloses third object 1030 using weather of user’s location, with data of first object used as obstacle such that snow appears around locations of GUI icons, with no snow within the icon itself);
Control the display to display the modified image and the UI object as the background screen by overlaying the UI object on the modified image (Park, ¶66: the processor of the electronic device may synthesize the second object based on the first object, e.g. in case that the first object is the home screen, and the second object is a live wallpaper screen, the live wallpaper may be synthesized based on the home screen, where the second data of the second object may also be synthesized with the first data based on the first object; Fig. 10A and ¶110 discloses synthesizing first object 1010 as overlay with third object as live wallpaper screen displayed on screen)
Park does not explicitly disclose the use of time in addition to weather as claimed.
Bialota, however, discloses:
Obtain an original image designated as the background screen (Bialota, ¶83: select user image as a base when a wallpaper is generated)
Obtain time information and weather information, (Bialota, ¶33: electronic device selects variable data that is reflected when composite image is created; ¶98: variable data includes external data acquired through web service; ¶100: external data acquired through the web service may include at least one of weather data, season data, time data, and place data; Also ¶101, weather, and ¶103, time; ¶111: composite image creating unit 114 may reflect a dynamic image of variable data related to weather in the object area. The composite image creating unit 114 may create a single composite image by composing the object area with the dynamic image of the variable data related to the weather when creating the image; ¶112: composite image creating unit 114 may reflect a dynamic image of variable data related to time in the object area, where composite image creating unit 114 may create a single composite image by composing the object area with the dynamic image of the variable data related to the time when creating the image)
Obtain, based on the time information and the weather information, a predefined prompt for image generation (Bialota, ¶34: the electronic device may create a composite image based on variable data and an object area acquired from a user image through image processing; ¶107: The composite image creating unit 114 may create a composite image by reflecting the selected object area and the selected variable data – i.e. prompt)
Generate a modified image corresponding to the time information and the weather information by applying the original image, the predefined prompt, and the display area information as input data to a generative model that generates the modified image (Bialota, ¶34: device may display the created composite image as a wallpaper through a display unit; ¶107: The composite image creating unit 114 may create a composite image by reflecting the selected object area and the selected variable data – i.e. prompt – including adding a dynamic image related to the variable data to the object data and changing brightness and colors; ¶109: composite image creating unit creates composite image by adding an image related to variable data to the object area, the dynamic image related to variable data that includes weather and time; ¶118: displaying wallpaper through display unit)
Both Park and Bialota are directed to systems and techniques for modifying user interfaces by adding a dynamic effect related to user context to a background image. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and technique of providing an interactive background for a user interface based on contextual data as provided by Park, by incorporating the additional contextual data for modifying the displayed images in an interactive user interface as provided by Bialota, using known electronic interfacing and programming techniques. The modification results in an improved contextual based user interface by allowing for more data related to the user’s environment and context to provide more interactivity and more relevant image information to the user.
Park modified by Bialota does not explicitly disclose the use of a generative model for the generation of the modified image (i.e. in place of the programmed model provided by Park)
Pardeshi discloses:
obtain display area information of a user interface (UI) object that is overlaid on the original image; (Pardeshi, ¶56: In at least one embodiment, a screen divider 318 can take as input text and asset-related conditions, and can learn whether any of these assets, test, or images can, or should, be scaled down or adjusted as necessary for a design layout for a generated output GUI 316, and can determine how much real estate is needed to render a particular interface, which can then be used to determine how many screens or pages may be required for an interface for a target platform; ¶61 further discloses analyzing GUI to identify visual features, including structure and design, as well as text elements; Fig. 2 also showing text tokens recognized separately from images, where text shown overlaid on input GUI 202)
Generate a modified image by applying the original image, the predefined prompt, and the display area information as input data to a generative model that generates the modified image, (Pardeshi ¶61: GUI analyzed to identify visual and functional features of the first GUI, including structure, design, color set, images, tokens and text elements, with these features transformed into feature vectors and encoded into a latent space for generation of second GUI, where context for elements or features of this first GUI can be determined 410 in order to determine relations of elements, such as may cause these elements to be kept together or able to be divided across multiple pages or screen of a generated interface; Also ¶62: determine one or more functional feature s of GUI and determine one or more visual or design features)
Park, Bialota and Pardeshi are directed to techniques for automatically modifying displayed computer graphical user interfaces to a user based on contextual data. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the technique and system for contextually modifying an interface element based on weather data using object layout and context data for augmenting the image as provided by Park, incorporating the additional contextual data for modifying the displayed images in an interactive user interface as provided by Bialota, by utilizing a machine learning model that parameterizes elements of a user interface and contextual information of the layout as input to autogenerate a modified user interface using machine learning as provided by Paredeshi, using known electronic interfacing and programming techniques. The modification merely substitutes a known algorithm for contextual modification of displayed graphical elements by using a machine learning model trained to modify displayed graphical elements based on device contextual data and element structure, which would have been a predictable substitution to automate the weighting and generation of image data based on parameterized data, such as element layout and weather context. Furthermore, the modification results in an improved arrangement of user interface elements by using a more robust machine learning system that provides enhanced learning and weighting of data to provide different results without requiring specific hardcoding, for easier implementation and more diverse utility and better arrangement of graphical elements for display.
The only remaining limitation not explicitly clear from the teachings of Park, Bialota and Pardeshi is that the first degree is less than the second degree such that in the modified image, the predefined prompt modifies the original image to a lesser degree in the display area than in the remaining area. (It is arguable that the teachings of Park include a first and second degree of displaying effects a based on the GUI elements with the first degree and second degree such that the first degree is less than the second degree, as the snow elements are shown in region around the GUI elements of Park, and not on the GUI elements, where no display of the effect is lesser than any display of the effect. However, as a result of conversation with applicant’s representative during interview, Kim is relied upon for showing that it was known at the time of the applicant’s invention at least some effect is displayed in “lesser” effect, based on identified objects and regions.)
Kim discloses:
wherein in the modified image, the predefined prompt is reflected in a display area where the UI object is displayed in accordance with a first degree, and the predefined prompt is reflected in a remaining area other than the display area in accordance with a second degree that is different from the first degree, wherein the first degree is less than the second degree such that in the modified image, the predefined prompt modifies the original image to a lesser degree in the display area than in the remaining area; (Kim, ¶54: device provides weather image effect to an image by cutting out a plurality of fragment images from a weather texture image and sequentially overlapping the plurality of fragment images onto the image, where device analyzes depth of objects within an image, and overlaps fragment images reflecting the current weather onto the image based on the analyzed depth, adjusting the transparency of fragment images according to depth and composite the transparent fragment images with adjusted transparency with the image to provide a weather effect on the original photo;
¶76: simulate image effect representing current weather eon the image using the acquired weather texture image, where device can determine cutting positions for weather texture image by portion within non texture image, where rain particles are displayed using a degree of shifting based on depth position of objects in image, including to display non-particles in near space as opposed in distance space; ¶79: discusses use of transparency based on depth;
¶107: device simulates weather effects on image based on reference information, including applying a predetermined transparency to fragment image according to masking of image corresponding to distant area; ¶¶109-110: weather texture image selected according to depth range, and determine positions of image fragments to be used for simulation of weather effects in weather texture image; ¶116: the server 2000 may determine criteria for simulating the weather texture image corresponding to the second depth rang having a depth deeper than that of the specific object in the image, and image segments obtained from the weather texture image corresponding to the second depth range may be simulated as if displayed behind the specific object, where the server 2000 may determine a shape of the image segments to be cut out from the weather texture image corresponding to the second depth range, in such a manner that the image segments do not overlap with the specific object in the image; ¶117 further discloses making the cut piece transparent or cutting the piece image to prevent overlap;
** Note ¶¶58-59, 66, 95, 98, and 104 and Fig. 6, discloses use of AI models for object recognition and segmentation for processing images to apply affects, including which weather texture image to use depending on the weather, criteria regarding which part of weather texture image to cut out for fragment image, spacing between fragment images and how long to display fragment image according to depth interval, although mainly highlighted to show integration with AI modeling, such as provided in previously cited art
)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the technique and system for contextually modifying an interface element based on weather data using object layout and context data for augmenting the image as provided by Park, incorporating the additional contextual data for modifying the displayed images in an interactive user interface as provided by Bialota, using a machine learning model that parameterizes elements of a user interface and contextual information of the layout as input to autogenerate a modified user interface using machine learning as provided by Paredeshi, by including the varying degree of emphasis of a effects based on different regions of the user interface as provided by Kim, using known electronic interfacing and programming techniques. The modification results in an improved context based user interface by providing more dynamic effects to show depth and allow foreground and background objects to be better differentiated such that elements can more easily be identified by a viewer for easier understanding and readability, while also providing more dynamic visual effects.
Regarding claim 1, the device of claim 11 performs the method of claim 1 and as such claim 1 is rejected based on the same rationale as claim 11 set forth above.
Regarding claim 12, Park modified by Bialota, Paredeshi and Kim further discloses:
obtain a parameter for adjusting the first degree or the second degree, and wherein the input data further includes the parameter. (Kim, ¶79: device 1000 may differently adjust transparencies of the image segments based on depths, i.e. parameter, and simulate the weather effect by using the transparency-adjusted image segments, e.g., when depths of spaces in the image range from 0 to 100, the device 1000 may adjust a transparency of an image segment corresponding to a depth from 0 to 40, to 30%, adjust a transparency of an image segment corresponding to a depth from 40 to 70, to 50%, and adjust a transparency of an image segment corresponding to a depth from 70 to 100, to 70; ¶¶58-59, 66, 95, 98, and 104 and Fig. 6, discloses use of AI models for object recognition and segmentation for processing images to apply affects, including which weather texture image to use depending on the weather, criteria regarding which part of weather texture image to cut out for fragment image, spacing between fragment images and how long to display fragment image according to depth interval)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the technique and system for contextually modifying an interface element based on weather data using object layout and context data for augmenting the image as provided by Park, incorporating the additional contextual data for modifying the displayed images in an interactive user interface as provided by Bialota, using a machine learning model that parameterizes elements of a user interface and contextual information of the layout as input to autogenerate a modified user interface using machine learning as provided by Paredeshi, by including the varying degree of emphasis of a effects based on different regions of the user interface as provided by Kim, using known electronic interfacing and programming techniques. The modification results in an improved context based user interface by providing more dynamic effects to show depth and allow foreground and background objects to be better differentiated such that elements can more easily be identified by a viewer for easier understanding and readability, while also providing more dynamic visual effects.
Regarding claim 2, the device of claim 12 performs the method of claim 2 and as such claim 2 is rejected based on the same rationale as claim 12 set forth above.
Regarding claim 14, Park further discloses:
obtain property information of the UI object, and change a property of the UI object, based on the property information and the modified image. (Park, Fig. 3 and ¶64: processor of electronic device identifies a first object and first data on the first object, where the first object may be an initial home screen of the electronic device, and may correspond to a screen including a plurality of layers, the first object may include three sheets of layers of the home screen and data (e.g., first data) of each layer, and the data of each layer is data about an application being displayed on each layer, and may include coordinates and attributes of widgets and icons (e.g., application name, application type, package information, and icon color; Fig. 5 and ¶84 discloses home screen composed of individual layers, where “the first data is data for the application being displayed on an individual layer, and may include coordinates of widgets and icons, and sizes or attributes of icons of the widgets and applications (e.g., application name, application type, package information, icon colors)”; ¶66: the processor of the electronic device may synthesize the second object based on the first object, e.g. in case that the first object is the home screen, and the second object is a live wallpaper screen, the live wallpaper may be synthesized based on the home screen, where the second data of the second object may also be synthesized with the first data based on the first object; Fig. 10A and ¶110 discloses synthesizing first object 1010 as overlay with third object as live wallpaper screen displayed on screen; ¶¶111-114 and Figs. 10A-10C further discloses the displaying of the overlaid modified UI object based on interaction with the animated background as an obstacle)
Park modified by Biolota and Pardeshi also discloses:
obtain property information of the UI object, and change a property of the UI object, based on the property information and the modified image (Pardeshi, ¶56: In at least one embodiment, a screen divider 318 can take as input text and asset-related conditions, and can learn whether any of these assets, test, or images can, or should, be scaled down or adjusted as necessary for a design layout for a generated output GUI 316, and can determine how much real estate is needed to render a particular interface, which can then be used to determine how many screens or pages may be required for an interface for a target platform; ¶61 further discloses analyzing GUI to identify visual features, including structure and design, as well as text elements; Fig. 2 also showing text tokens recognized separately from images, where text shown overlaid on input GUI 202)
Park, Bialota and Pardeshi are combinable for the same reasons set forth above.
Regarding claim 4, the device of claim 14 performs the method of claim 4 and as such claim 4 is rejected based on the same rationale as claim 14 set forth above.
Regarding claim 15, Park further discloses:
identify, based on the property information, an image feature of the display area of the UI object in the modified image, and change the property of the UI object based on the image feature of the display area. (Park, Fig. 3 and ¶64: coordinates and attributes of widgets and icons (e.g., application name, application type, package information, and icon color; Fig. 5 and ¶84 discloses home screen composed of individual layers, where “the first data is data for the application being displayed on an individual layer, and may include coordinates of widgets and icons, and sizes or attributes of icons of the widgets and applications (e.g., application name, application type, package information, icon colors)”; ¶66: the processor of the electronic device may synthesize the second object based on the first object, e.g. in case that the first object is the home screen, and the second object is a live wallpaper screen, the live wallpaper may be synthesized based on the home screen, where the second data of the second object may also be synthesized with the first data based on the first object; Fig. 10A and ¶110 discloses synthesizing first object 1010 as overlay with third object as live wallpaper screen displayed on screen; ¶¶111-114 and Figs. 10A-10C further discloses the displaying of the overlaid modified UI object based on interaction with the animated background as an obstacle)
identify, based on the property information, an image feature of the display area of the UI object in the modified image, and change the property of the UI object based on the image feature of the display area (Pardeshi, ¶56: In at least one embodiment, a screen divider 318 can take as input text and asset-related conditions, and can learn whether any of these assets, test, or images can, or should, be scaled down or adjusted as necessary for a design layout for a generated output GUI 316, and can determine how much real estate is needed to render a particular interface, which can then be used to determine how many screens or pages may be required for an interface for a target platform; ¶61 further discloses analyzing GUI to identify visual features, including structure and design, as well as text elements; Fig. 2 also showing text tokens recognized separately from images, where text shown overlaid on input GUI 202)
Park, Bialota and Pardeshi are combinable for the same reasons set forth above.
Regarding claim 5, the device of claim 15 performs the method of claim 5 and as such claim 5 is rejected based on the same rationale as claim 15 set forth above.
Regarding claim 16, Park modified by Bialota and Pardeshi further discloses:
wherein the predefined prompt comprises a first predefined prompt and a second predefined prompt, and wherein the one or more instructions, when executed by the one or more processors, further cause the electronic device to generate, by using the generative model, the modified image such that the first predefined prompt is reflected in a first region of the original image, and the second predefined prompt is reflected in a second region of the original image (Pardeshi, ¶56: GUI analyzed to identify visual and functional features of the first GUI, including structure, design, color set, images, tokens and text elements, with these features transformed into feature vectors and encoded into a latent space for generation of second GUI, where context for elements or features of this first GUI can be determined 410 in order to determine relations of elements, such as may cause these elements to be kept together or able to be divided across multiple pages or screen of a generated interface; Note Figs. 1 and 2 showing elements that are turned into prompts for generator 242)
Park, Bialota and Pardeshi are combinable for the same reasons set forth above.
Regarding claim 6, the device of claim 16 performs the method of claim 6 and as such claim 6 is rejected based on the same rationale as claim 16 set forth above.
Regarding claim 8, Park further discloses displaying the modified image changing overtime with the display of the UI object in an overlaid manner (see Park, ¶67 discloses invention as a live wallpaper that includes animation effects over time; Also Fig. 10A and ¶110 discussing snowy particles to reflect weather)
Park modified by Bialota further discloses:
Wherein the displaying the modified image and the object as the background screen by overlaying the object on the modified image comprises dynamically change the modified image and the object based on the time information and weather information and displaying a result of the changing on the background screen. (Bialota, ¶34: device may display the created composite image as a wallpaper through a display unit; ¶107: The composite image creating unit 114 may create a composite image by reflecting the selected object area and the selected variable data – i.e. prompt – including adding a dynamic image related to the variable data to the object data and changing brightness and colors; ¶109: composite image creating unit creates composite image by adding an image related to variable data to the object area, the dynamic image related to variable data that includes weather and time; ¶118: displaying wallpaper through display unit)
Both Park and Bialota are directed to systems and techniques for modifying user interfaces by adding a dynamic effect related to user context to a background image. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and technique of providing an interactive background for a user interface based on contextual data as provided by Park, by incorporating the additional contextual data for modifying the displayed images in an interactive user interface as provided by Bialota, using known electronic interfacing and programming techniques. The modification results in an improved contextual based user interface by allowing for more data related to the user’s environment and context to provide more interactivity and more relevant image information to the user.
Claim(s) 7 and 17-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over
Park et al. (US 2022/0058038 A1) in view of
Bialota (US 2015/0029206 A1)
Pardeshi et al. (US 2021/0406673 A1) and
Kim et al. (US 2020/0265616 A1) in further view of
Hackborn et al. (US 2011/0119610 A1).
Regarding claim 17, the limitations included from claim 11 are rejected based on the same rationale as claim 11 set forth above. Further regarding claim 17, Hackborn discloses:
generate contextual text related to the time information and the weather information, and control the display to display the contextual text on a screen of the electronic device (Hackborn, Fig. 5B and ¶44: A textual indication 539 of the weather conditions is displayed ("Snow Showers"), along with a current temperature 542 and forecasted high and low temperatures 544. The weather information may be received from another computing device, for example, and the code associated with the wallpaper object may cause the information to be displayed on the wallpaper)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the technique and system for contextually modifying an interface element based on weather data using object layout and context data for augmenting the image as provided by Park, incorporating the additional contextual data for modifying the displayed images in an interactive user interface as provided by Bialota, using a machine learning model that parameterizes elements of a user interface and contextual information of the layout as input to autogenerate a modified user interface using machine learning as provided by Paredeshi, and including the varying degree of emphasis of a effects based on different regions of the user interface as provided by Kim, by including the additional data as provided by Hackborn, using known electronic interfacing and programming techniques. The modification results in an improved context based user interface by providing additional readable data for easier understanding by a user and providing more relevant data otherwise not easily indicated in visual form (e.g. precise temperature).
Regarding claim 7, the device of claim 17 performs the method of claim 7 and as such claim 7 is rejected based on the same rationale as claim 17 set forth above.
Regarding claim 18, Park further discloses displaying the modified image changing overtime with the display of the UI object in an overlaid manner on the screen of the electronic device (see Park, ¶67 discloses invention as a live wallpaper that includes animation effects over time; Also Fig. 10A and ¶110 discussing snowy particles to reflect weather)
Park modified by Bialota further discloses:
Control the display to dynamically change the modified image and the object based on the time information and the weather information and displaying a result of the changing on the screen (Bialota, ¶34: device may display the created composite image as a wallpaper through a display unit; ¶107: The composite image creating unit 114 may create a composite image by reflecting the selected object area and the selected variable data – i.e. prompt – including adding a dynamic image related to the variable data to the object data and changing brightness and colors; ¶109: composite image creating unit creates composite image by adding an image related to variable data to the object area, the dynamic image related to variable data that includes weather and time; ¶118: displaying wallpaper through display unit)
Both Park and Bialota are directed to systems and techniques for modifying user interfaces by adding a dynamic effect related to user context to a background image. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and technique of providing an interactive background for a user interface based on contextual data as provided by Park, by incorporating the additional contextual data for modifying the displayed images in an interactive user interface as provided by Bialota, using known electronic interfacing and programming techniques. The modification results in an improved contextual based user interface by allowing for more data related to the user’s environment and context to provide more interactivity and more relevant image information to the user.
Claim(s) 9-10 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over
Park et al. (US 2022/0058038 A1) in view of
Bialota (US 2015/0029206 A1)
Pardeshi et al. (US 2021/0406673 A1) and
Kim et al. (US 2020/0265616 A1) in further view of
Suzuki et al. (US 2020/0160575 A1)
Regarding claim 19, the limitations included from claim 11 are rejected based on the same rationale as claim 11 set forth above. Further regarding claim 19, Pardeshi further discloses:
by inputting the original image into the generative model, each synthetic image from the plurality of synthetic images comprising at least one feature of the original image (Pardeshi, ¶49 and ¶¶51-52: discloses using a generative adversarial network for component selection and translated GUI generation, where generator generates at least one screen of page for an output GUI and screen determination can be fed as a constraint to generator 242 for generating output GUI 316, or in at least one embodiment may be passed to encoder 232 for encoding into a latent space for input as a constraint to generator 242; determining a combination of primary colors for elements of an interface, as well as how those colors are utilized in this design, such as for specific elements or types of elements, or for patterns of color application; ¶57 further discloses the use of one or more GANs for generation of images to output design GUI layouts based on constraints provided as inputs, including GAN to generate one or more screens for an output GUI; ¶61 further discloses the transform of a first GUI using identified visual and functional features of the first GUI related to structure, design, color set, text elements,, which are transformed into feature vectors, encoded in latent space and used to generate second GUI, where “images or other content generated for one or more second GUIs can then be provided 412 as output from this translation, where the second GUIs retain a general behavior or functionality of this first GUI while also retaining various design or visual features of that GUI.”)
Park, Bialota, and Pardeshi are directed to techniques for automatically modifying displayed computer graphical user interfaces to a user based on contextual data. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the technique and system for contextually modifying an interface element based on weather data using object layout and context data for augmenting the image as provided by Park, incorporating the additional contextual data for modifying the displayed images in an interactive user interface as provided by Bialota, by utilizing a machine learning model that parameterizes elements of a user interface and contextual information of the layout as input to autogenerate a modified user interface using machine learning as provided by Paredeshi, using known electronic interfacing and programming techniques. The modification merely substitutes a known algorithm for contextual modification of displayed graphical elements by using a machine learning model trained to modify displayed graphical elements based on device contextual data and element structure, which would have been a predictable substitution to automate the weighting and generation of image data based on parameterized data, such as element layout and weather context. Furthermore, the modification results in an improved arrangement of user interface elements by using a more robust machine learning system that provides enhanced learning and weighting of data to provide different results without requiring specific hardcoding, for easier implementation and more diverse utility and better arrangement of graphical elements for display.
Although it is well known to generate a plurality of images using generative models as provided by Pardeshi, particularly inherent in the training of the models, Pardeshi does not explicitly disclose the selection from the plurality of generated images.
Suzuki discloses:
obtain, by inputting the original image into the generative model, a plurality of synthetic images as a plurality of candidate images, each synthetic image from the plurality of synthetic image comprising at least one feature of the original image, and determine the modified image by selection of a candidate image from among the plurality of candidate images (Suzuki, ¶22: compensated image generator may generate at least one compensated preference parameter based on a current preference parameter, and the compensated image generator may generate a plurality of compensated images of the input image based on the at least one generated compensated preference parameter, and the image selector may display the plurality of compensated images and outputs a compensated image that has been selected by a user as an output image from the plurality of displayed compensated images; Fig. 7 shows compensated images with feature from original image Fig. 2A; ¶56: The compensated image generator 120 generates a plurality of compensated images (COMPENSATED IMAGES (CI)) by compensating the input image II based on the compensated preference parameters CPP; ¶60: The image selector 130 displays the compensated images CI to a user. The image selector 130 outputs a selected compensated image, which is selected by the user the compensated images CI, as an output image (OUTPUT IMAGE (OI)).)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the technique and system for contextually modifying an interface element based on weather data using object layout and context data for augmenting the image as provided by Park, incorporating the additional contextual data for modifying the displayed images in an interactive user interface as provided by Bialota, utilizing a machine learning model that parameterizes elements of a user interface and contextual information of the layout as input to autogenerate a modified user interface using machine learning as provided by Paredeshi, including the varying degree of emphasis of a effects based on different regions of the user interface as provided by Kim, by including the technique for presenting a plurality of generated images from an automated image modification model for user selection as provided by Suzuki, using known electronic interfacing and programming techniques. The modification results in an improved user interface modification by allowing a user greater control over the results by presenting the user with multiple options and allowing user selection of the preferred result.
Regarding claim 9, the device of claim 19 performs the method of claim 9 and as such claim 9 is rejected based on the same rationale as claim 19 set forth above.
Regarding claim 10, Park further discloses user input for selecting data related with the user interface (Park, ¶68: direct input may include an input using a touch panel included in the electronic device using a user's body or tool; ¶134: user performs input for selecting data)
Pardeshi further discloses:
Wherein the determining the modified image comprises determining the modified image based on user input (Suzuki, ¶60: display compensated images to user, where user selects an image; input/output device 350 including keyboard, mouse)
Park is combinable with Bialota, Pardeshi, Kim and Suzuki for the same reasons set forth above.
Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over
Park et al. (US 2022/0058038 A1) in view of
Bialota (US 2015/0029206 A1) and in further view of
Pardeshi et al. (US 2021/0406673 A1).
Regarding claim 20, Park discloses:
A method, performed by an electronic device, of changing a background screen, the method comprising: (Park, Abstract and ¶9: configuring background screen of electronic device)
obtaining an original image designated as the background screen; (Park, Fig. 7A and ¶¶93-95 discloses second object corresponding to dynamic wall paper having layered objects, including e.g. b-th object corresponding to image content and a-th corresponding to particle content, together an “original image”; ¶110: second object corresponds to live wallpaper screen)
obtaining(Park, ¶110: specific second data may be determined depending on the weather reflecting a user's location)
obtaining, for the original image based on the time information and the weather information, a predefined prompt of a generative model; (Park, ¶95: particle content for snowy weather; ¶110: specific second data may be determined depending on weather reflecting a user’s location, e.g. if weather reflecting user’s location is “snowy”, the second data may correspond to display of “snowy” as a particle)
obtaining property information of a user interface (UI) object that is overlaid on the original image; (Park, Fig. 3 and ¶64: processor of electronic device identifies a first object and first data on the first object, where the first object may be an initial home screen of the electronic device, and may correspond to a screen including a plurality of layers, the first object may include three sheets of layers of the home screen and data (e.g., first data) of each layer, and the data of each layer is data about an application being displayed on each layer, and may include coordinates and attributes of widgets and icons (e.g., application name, application type, package information, and icon color; Fig. 5 and ¶84 discloses home screen composed of individual layers, where “the first data is data for the application being displayed on an individual layer, and may include coordinates of widgets and icons, and sizes or attributes of icons of the widgets and applications (e.g., application name, application type, package information, icon colors)”)
generating a modified image corresponding to (Park, ¶¶95-96 discloses the processor analyzing the objects such that the synthesis of the a-th to f-th objects may be changed; Fig 10A and ¶110 discloses third object 1030 using weather of user’s location, with data of first object used as obstacle) and
changing a property of the UI object based on the changed property information of the UI object (Park, ¶110 discusses UI elements as obstacles for the changing interface; Fig. 10B and ¶¶111-112: processor may determine whether to activate first data of the first object 1010 based on the synthesis of the third object, where “if it is determined that the first data 1011 is activated, the processor may be configured to recognize the icon of the first data as an obstacle by performing the second object (e.g., by executing the second data 1021) on the third object 1030 when the predetermined event is detected. The processor may be configured to map a control (e.g., a user's control) of the second data 1021 for the content constituting the second object onto the third object 1030 and the third data”; ¶113: f it is determined that the first data 1011 is inactivated, the processor may be configured to recognize as if the icon of the first data does not exist by performing the second object (e.g., by executing the second data 1021) on the third object 1030 when the predetermined event is detected. The icon of the first data may be expressed on the third object in a manner that the icon is displayed blurry or in a dotted line; ¶114 discloses when active state determined; the processor may execute the second data of the second object on the third object, and may configure to render the third object depending on whether the first data is activated. According to an embodiment, if the first data is determined to be in an active state, the processor may add the first data to the second data to be executed during the execution of the second data on the third object. The addition of the first data to the second data to be executed may correspond to that the display of the icon of the first data is rendered as an obstacle of the second data (e.g., obstacle as the contents of the game application) (e.g., FIG. 10B).); and
displaying, after the changing the property of the UI object, the modified image and the UI object on a screen of the electronic device by overlaying the UI object on the modified image. (Park, ¶66: the processor of the electronic device may synthesize the second object based on the first object, e.g. in case that the first object is the home screen, and the second object is a live wallpaper screen, the live wallpaper may be synthesized based on the home screen, where the second data of the second object may also be synthesized with the first data based on the first object; Fig. 10A and ¶110 discloses synthesizing first object 1010 as overlay with third object as live wallpaper screen displayed on screen of electronic device; ¶¶111-114 and Figs. 10A-10C further discloses the displaying of the overlaid modified UI object based on interaction with the animated background as an obstacle)
Park does not explicitly disclose the use of time in addition to weather as claimed.
Bialota, however, discloses:
Obtain an original image designated as a background screen (Bialota, ¶83: select user image as a base when a wallpaper is generated)
Obtain time information and weather information, (Bialota, ¶33: electronic device selects variable data that is reflected when composite image is created; ¶98: variable data includes external data acquired through web service; ¶100: external data acquired through the web service may include at least one of weather data, season data, time data, and place data; Also ¶101, weather, and ¶103, time; ¶111: composite image creating unit 114 may reflect a dynamic image of variable data related to weather in the object area. The composite image creating unit 114 may create a single composite image by composing the object area with the dynamic image of the variable data related to the weather when creating the image; ¶112: composite image creating unit 114 may reflect a dynamic image of variable data related to time in the object area, where composite image creating unit 114 may create a single composite image by composing the object area with the dynamic image of the variable data related to the time when creating the image)
Obtain, based on the time information and the weather information, a predefined prompt for image generation (Bialota, ¶34: the electronic device may create a composite image based on variable data and an object area acquired from a user image through image processing; ¶107: The composite image creating unit 114 may create a composite image by reflecting the selected object area and the selected variable data – i.e. prompt)
Generate a modified image corresponding to the time information and the weather information by applying the original image, the predefined prompt, and the display area information as input data to a generative model that generates the modified image, (Bialota, ¶34: device may display the created composite image as a wallpaper through a display unit; ¶107: The composite image creating unit 114 may create a composite image by reflecting the selected object area and the selected variable data – i.e. prompt – including adding a dynamic image related to the variable data to the object data and changing brightness and colors; ¶109: composite image creating unit creates composite image by adding an image related to variable data to the object area, the dynamic image related to variable data that includes weather and time; ¶118: displaying wallpaper through display unit)
Both Park and Bialota are directed to systems and techniques for modifying user interfaces by adding a dynamic effect related to user context to a background image. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and technique of providing an interactive background for a user interface based on contextual data as provided by Park, by incorporating the additional contextual data for modifying the displayed images in an interactive user interface as provided by Bialota, using known electronic interfacing and programming techniques. The modification results in an improved contextual based user interface by allowing for more data related to the user’s environment and context to provide more interactivity and more relevant image information to the user.
Park modified by Bialota does not explicitly disclose the use of a generative model for the generation of the modified image (i.e. in place of the programmed model provided by Park)
Pardeshi discloses:
obtain display area information of a user interface (UI) object that is overlaid on the original image; (Pardeshi, ¶56: In at least one embodiment, a screen divider 318 can take as input text and asset-related conditions, and can learn whether any of these assets, test, or images can, or should, be scaled down or adjusted as necessary for a design layout for a generated output GUI 316, and can determine how much real estate is needed to render a particular interface, which can then be used to determine how many screens or pages may be required for an interface for a target platform; ¶61 further discloses analyzing GUI to identify visual features, including structure and design, as well as text elements; Fig. 2 also showing text tokens recognized separately from images, where text shown overlaid on input GUI 202)
Generate a modified image by applying the original image, the predefined prompt, and the display area information as input data to a generative model that generates the modified image, (Pardeshi ¶61: GUI analyzed to identify visual and functional features of the first GUI, including structure, design, color set, images, tokens and text elements, with these features transformed into feature vectors and encoded into a latent space for generation of second GUI, where context for elements or features of this first GUI can be determined 410 in order to determine relations of elements, such as may cause these elements to be kept together or able to be divided across multiple pages or screen of a generated interface; Also ¶62: determine one or more functional feature s of GUI and determine one or more visual or design features)
changing a property of the UI object based on the changed property information of the UI object, the changed property information obtained from the generative model (Pardeshi, ¶56: In at least one embodiment, a screen divider 318 can take as input text and asset-related conditions, and can learn whether any of these assets, test, or images can, or should, be scaled down or adjusted as necessary for a design layout for a generated output GUI 316, and can determine how much real estate is needed to render a particular interface, which can then be used to determine how many screens or pages may be required for an interface for a target platform; ¶61 further discloses analyzing GUI to identify visual features, including structure and design, as well as text elements, color set, and tokens, and transforming the features into vectors encoded in latent space defining constraints for generation of a second GUI, which infers an image having appropriate layout for each page of the second GUI; Fig. 2 also showing text tokens recognized separately from images, where text shown overlaid on input GUI 202)
Park, Bialota and Pardeshi are directed to techniques for automatically modifying displayed computer graphical user interfaces to a user based on contextual data. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the technique and system for contextually modifying an interface element based on weather data using object layout and context data for augmenting the image as provided by Park, incorporating the additional contextual data for modifying the displayed images in an interactive user interface as provided by Bialota, by utilizing a machine learning model that parameterizes elements of a user interface and contextual information of the layout as input to autogenerate a modified user interface using machine learning as provided by Paredeshi, using known electronic interfacing and programming techniques. The modification merely substitutes a known algorithm for contextual modification of displayed graphical elements by using a machine learning model trained to modify displayed graphical elements based on device contextual data and element structure, which would have been a predictable substitution to automate the weighting and generation of image data based on parameterized data, such as element layout and weather context. Furthermore, the modification results in an improved arrangement of user interface elements by using a more robust machine learning system that provides enhanced learning and weighting of data to provide different results without requiring specific hardcoding, for easier implementation and more diverse utility and better arrangement of graphical elements for display.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action. Any inquiry concerning this communication or earlier communications from the examiner should be directed to WILLIAM A BEUTEL whose telephone number is (571)272-3132. The examiner can normally be reached Monday-Friday 9:00 AM - 5:00 PM (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, DANIEL HAJNIK can be reached at 571-272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WILLIAM A BEUTEL/Primary Examiner, Art Unit 2616
Prosecution Insights