Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
2. This action is responsive to the Application filed on 9/25/2024. A filing date 9/25/2024 is acknowledged. The sought benefit of provisional application 63631392 (which was filed on 4/8/2024), provisional application 6364834 (which was filed on 5/10/2024), provisional application 63657810 (which was filed on 6/8/2024), provisional application 63692153 (which was filed on 9/8/2024), is acknowledged. Claims 1-28 are pending in this application. Claims 1, 27, 28 are independent claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
3. Claims 1-28 are rejected under 35 U.S.C. 103 as being unpatentable over Matan Stauber et al (US Publication 20200105003 A1, hereinafter Stauber), and in view of Archie Bagnall (US Publication 20240361891 A1, hereinafter Bagnall).
As for independent claim 1, Stauber discloses: A computer system configured to communicate with one or more display generation components and one or more input devices (Abstract, The present disclosure generally relates to displaying and editing image with depth information, comprising: one or more processors; and
memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for ([0007], one or more processors of an electronic device with a display. The one or more programs include instructions for: receiving a request to display an image that includes a subject, wherein image data associated with the image includes depth information associated with the subject): while displaying, via the one or more display generation components, a media item including a foreground element and a first background element ([0228], a user is in the foreground, and background elements, such as a tree positioned behind the user, are in the background), detecting, via the one or more input devices, an input corresponding to a request to modify the media item ([0011], receiving a request to edit a portion of the image), including:
detecting at least a portion of the input is directed to a location of the foreground element and a location of the first background element ([0177], the device detects contacts (e.g., 460 and 462 in FIG. 4B) with the touch-sensitive surface 451 at locations that correspond to respective locations on the display; [0247], While displaying image 906 in user interface 904, an input 903 (e.g., a finger contact) is detected at a location on image 906 corresponding to background element 908b); and in response to detecting the input corresponding to the request to modify the media item:
in accordance with a determination that an initial location of the input is directed to the location of the first background element ([0247], While displaying image 906 in user interface 904, an input 903 (e.g., a finger contact) is detected at a location on image 906 corresponding to background element 908b), modifying the media item, that is displayed via the one or more display generation components, by modifying at least a portion of the first background element without modifying the foreground element to which the input was directed ([0269], The electronic device displays the isolated object separately from (e.g., as a cut out, in isolation without displaying the one or more other elements) the one or more other of elements in the image. In some examples, the object (e.g., 922) is displayed and edited independently of the rest of the image; [0271], Modifying visual characteristics in a selected depth range without modifying visual characteristics outside the selected depth range allows a user to edit an image more efficiently; [0274], in response to receiving the second request to edit the second portion of the image, the electronic device (e.g., 900) modifies (1010), based on the depth information, visual characteristics of one or more second elements (e.g., a wall, a face, a location in space (e.g., where text is to be inserted)) of the plurality of elements in a second depth range without modifying visual characteristics of elements that are not in (e.g., that are outside of) the second depth range. Modifying visual characteristics in a selected depth range without modifying visual characteristics outside the selected depth range allows a user to edit an image more efficiently).
Stauber discloses editing image with depth information but does not clearly disclose determining an initial location of the input, in an analogous art of editing image with depth information, Bagnall discloses: an initial location of the input (Bagnall: [0572], determining a location of the input element 4602a (and the corresponding three-dimensional object) within the three-dimensional space, the scene-based image editing system 106 determines the focal point for the modified two-dimensional image 4600a);
Stauber and Bagnall are analogous arts because they are in the same field of endeavor, editing image with depth information. Therefore, it would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the invention of Stauber using the teachings of Bagnall to include determining the focal point of the input location. It would provide Graham’s method with enhanced capabilities of allowing user to modify the selected portion without compromising the integrity of the surrounding portions of the displacement three-dimensional mesh more efficiently (Bagnall: [0541]).
As for claim 2, Stauber-Bagnall discloses: in response to detecting the input corresponding to the request to modify the media item: in accordance with a determination that the initial location of the input is directed to the location of the foreground element, modifying the media item by modifying at least a portion of the foreground element without modifying the first background element to which the input was directed (Stauber: [0011], in response to receiving the request to edit the portion of the image, modifying, based on the depth information, visual characteristics of one or more elements of the plurality of elements in a first depth range without modifying visual characteristics of elements that are not in the first depth range).
As for claim 3, Stauber-Bagnall discloses: detecting at least a portion of the input is directed to a location of the foreground element at a first time; detecting at least a portion of the input is directed to a location that does not correspond to a location of the foreground element at a second time, after the first time; and
detecting at least a portion of the input is directed to a location of the foreground element at a third time, after the second time (Stauber: ([0177], the device detects contacts (e.g., 460 and 462 in FIG. 4B) with the touch-sensitive surface 451 at locations that correspond to respective locations on the display; [0252], while displaying image 906 in user interface 904, an input 905 (e.g., a finger contact) is detected at a location on image 906 corresponding to foreground element 908a; please note the input is detected and the corresponding location is determined, including the input location is directed to the foreground element and moved away and then moved back).
As for claim 4, Stauber-Bagnall discloses: detecting at least a portion of the input is directed to a location of a second background element, different from the first background element (Bagnall: [0113], The background for an object, in or more embodiments, comprises multiple object/semantic areas. For example, the background for an object can comprise part of another object and part of the furthest element in the image; please note the background comprise a first background element and another background element).
As for claim 5, Stauber-Bagnall discloses: replacing at least the portion of the first background element with automatically-generated visual content that is based on content of the media item that is near the first background element (Stauber: [0248], As shown in FIG. 9B, in response to detecting input 903, a graphical object 910 (e.g., text) is inserted into image 906 at a depth corresponding to the depth range of input 903).
As for claim 6, Stauber-Bagnall discloses: wherein the foreground element and the first background element are distinguished from each other automatically based on information associated with the media item (Stauber: [0246], Image 906 includes a foreground element 908a (e.g., a person) and a background element 908b (e.g., a wall) … A depth map (e.g., depth map image) contains information (e.g., values) that relates to the distance of objects in a scene (e.g., foreground element 908a, background element 908b)).
As for claim 7, Stauber-Bagnall discloses: prior to detecting the input corresponding to the request to modify the media item, detecting a request to display a media editing user interface (Stauber: Abstract, a request to display the image); and
in response to detecting the request to display the media editing user interface, displaying, via the one or more display generation components, a visual effect emphasizing a first type of element of the media item relative to a second type of element of the media item, different from the first type of element (Stauber: [0228], This distinction can be exploited to, for example, apply different visual effects (e.g., visual effects having a depth component) to the subject and background).
As for claim 8, Stauber-Bagnall discloses: wherein emphasizing the first type of element includes decreasing the visual effect that emphasizes of the first type of element relative to the second type of element over a period of time (Stauber: [0227], Displaying an image with simulated lighting improves visual feedback by enabling a user to more easily recognize the subject of the image (e.g., by emphasizing the contours of the subject's face); [0270], the electronic device receives an input instructing how the image is to be edited (e.g., dragging a slider after selecting a particular person to apply a color filter to that person or increase/decrease saturation of that person)).
As for claim 9, Stauber-Bagnall discloses: wherein emphasizing the first type of element relative to the second type of element includes displaying a visual indication having a brightness level that includes a degree of brightness that exceeds a standard dynamic range of brightness for the one or more display generation components (Stauber: [0281], the indication (e.g., 914) also indicates the brightness level of the simulated lighting (e.g., with the size of the icon, brightness of the icon, or additional visual elements)).
As for claim 10, Stauber-Bagnall discloses: wherein modifying the first background element includes modifying a respective portion of the first background element, wherein the input is not directed to the respective portion of the first background element (Stauber: [0250], the location of inserted graphical object 910 is modified in response to detecting a change in orientation (e.g., with an accelerometer and/or gyroscope) of the electronic device 900. For example, the location of inserted graphical object 910 can be modified as the orientation of electronic device 900 changes to create a parallax between graphical object 910 and other elements of image 906).
As for claim 11, Stauber-Bagnall discloses: wherein detecting the input corresponding to the request to modify the media item includes detecting a movement component of the input directed to one or more locations of the first background element (Stauber: ([0177], the device detects contacts (e.g., 460 and 462 in FIG. 4B) with the touch-sensitive surface 451 at locations that correspond to respective locations on the display; [0247], While displaying image 906 in user interface 904, an input 903 (e.g., a finger contact) is detected at a location on image 906 corresponding to background element 908b; please note the input is detected and the corresponding location is determined, including the movement of the input).
As for claim 12, Stauber-Bagnall discloses: in response to detecting the input corresponding to the request to modify the media item: displaying, via the one or more display generation components, a first indication at one or more locations corresponding to one or more locations of the movement component (Bagnall: [0553], the scene-based image editing system 106 generates the three-dimensional object (e.g., an orb, a cube, a plane, a point) in the three-dimensional representation 4502 in response to an initial input or request to set or modify a focal point of the two-dimensional image 4500); and gradually reducing a visual emphasis of the first indication over time (Bagnall: [0369], the scene-based image editing system 106 utilizes the object modification neural network 1806 to modify the degree to which an object attribute (or other attribute of a semantic area) appears (e.g., making a red apple less red or increasing the brightness of a digital image)).
As for claim 13, Stauber-Bagnall discloses: wherein the first indication includes a first visual characteristic, and wherein the one or more programs further include instructions for: in response to detecting the input corresponding to the request to modify the media item and in accordance with a determination that an initial location of the input is directed to the location of the first background element:
displaying, via the one or more display generation components, an indication at one or more locations of the first background element, wherein the indication at one or more locations of the first background element includes the first visual characteristic (Stauber: [0256], A position of the simulated light source is indicated by light icon 914. In some embodiments, the size of light icon 914 further indicates the intensity (e.g., brightness) of the simulated light source).
As for claim 14, Stauber-Bagnall discloses: wherein the first indication is displayed with a brightness level that includes a degree of brightness that exceeds a standard dynamic range of brightness for the one or more display generation components (Stauber: [0281], the indication (e.g., 914) also indicates the brightness level of the simulated lighting (e.g., with the size of the icon, brightness of the icon, or additional visual elements)).
As for claim 15, Stauber-Bagnall discloses: in response to detecting the input corresponding to the request to modify the media item, wherein the input corresponding to the request to modify the media item traverses a path:
in accordance with a determination that the path crosses one or more detected visual elements of the media item, displaying a first type of indication; and
in accordance with a determination that the path does not cross one or more detected visual elements of the media item, displaying a second type of indication, different from the first type of indication (Bagnall: [0220], the scene-based image editing system 106 draws a box that grows with the dragging motion and selects all objects that falls within the box. In some cases, the scene-based image editing system 106 draws a line that follows the path of the dragging motion and selects all objects intercepted by the line).
As for claim 16, Stauber-Bagnall discloses: while displaying the media item including the first element, detecting, via the one or more input devices, a respective input directed to a location of the first element and without detecting a movement component of the respective input; and in response to detecting the respective input directed to the location of the first element and without detecting the movement component of the respective input, selecting the first element of the media item (Stauber: [0177], the device detects contacts (e.g., 460 and 462 in FIG. 4B) with the touch-sensitive surface 451 at locations that correspond to respective locations on the display; [0247], While displaying image 906 in user interface 904, an input 903 (e.g., a finger contact) is detected at a location on image 906 corresponding to background element 908b).
As for claim 17, Stauber-Bagnall discloses: in response to detecting the input corresponding to the request to modify the media item and in accordance with the determination that the initial location of the input is directed to the location of the first background element: displaying, via the one or more display generation components, a visual indication that visually distinguishes the first background element from content in the media item (Stauber: Fig. 9A, [0249], a focal plane of the depth range of input 903).
As for claim 18, Stauber-Bagnall discloses: in response to detecting the input corresponding to the request to modify the media item and in accordance with the determination that the initial location of the input is directed to the location of the first background element: ceasing display, via the one or more display generation components, of a visual indication that visually distinguishes a second element from content of the media item (Stauber: Fig. 9B).
As for claim 19, Stauber-Bagnall discloses: wherein modifying at least a portion of the first background element without modifying the foreground element to which the input was directed includes: in accordance with a determination that the media item is displayed at a first zoom level, modifying at least the portion of the first background element using a respective dimension for modifying the media item in response to detecting an input; and in accordance with a determination that the media item is displayed at a second zoom level, different from the first zoom level, modifying at least the portion of the first background element using the respective dimension for modifying the media item in response to detecting the input (Stauber: [0256], In some embodiments, an intensity of the simulated light source (e.g., brightness) is changed based on the characteristic intensity of input 907. In some embodiments, a depth of the simulated light source (e.g., z position) is changed based on the characteristic intensity of input 907).
As for claim 20, Stauber-Bagnall discloses: wherein modifying at least a portion of the first background element without modifying the foreground element to which the input was directed includes: in accordance with a determination that a setting for a dimension for modifying the media item in response to detecting a respective input is set at a first value, modifying at least the portion of the first background element using a first dimension for the respective input; and
in accordance with a determination that the setting for the dimension for modifying the media item in response to detecting the respective input is set at a second value that is different from the first value, modifying at least the portion of the first background element using a second dimension for the respective input that is different from the first dimension (Stauber: [0256], In some embodiments, an intensity of the simulated light source (e.g., brightness) is changed based on the characteristic intensity of input 907. In some embodiments, a depth of the simulated light source (e.g., z position) is changed based on the characteristic intensity of input 907).
As for claim 21, Stauber-Bagnall discloses: modifying at least the portion of the first background element includes displaying, via the one or more display generation components, an animation of gradually ceasing to display the portion of the first background element, including: in accordance with a determination that the portion of the first background element has a first color, a respective color of the animation is based on the first color; and in accordance with a determination that the portion of the first background element has a second color, different from the first color, the respective color of the animation is based on the second color (Stauber: [0194], the animation progresses in accordance with an intensity of contact; [0207], as the animation progresses, different portions of image 706 are lit with different levels of simulated lighting, due to the contours of the face of the person in image 706 and a perceived motion of simulated light source(s) during the reveal animation; [0209], at the end of the reveal animation (as shown in FIG. 7C), image 706 is displayed with simulated lighting based on a respective (e.g., current) time of day (e.g., brightness, color, angle with respect to subject are modified based on the time of day (e.g., less light on subject at night time)).
As for claim 22, Stauber-Bagnall discloses: wherein the animation of gradually ceasing display of the portion of the first background element includes moving a graphical element having the respective color away from the portion of the first background element (Stauber: [0194], the animation progresses in accordance with an intensity of contact; [0207], as the animation progresses, different portions of image 706 are lit with different levels of simulated lighting, due to the contours of the face of the person in image 706 and a perceived motion of simulated light source(s) during the reveal animation).
As for claim 23, Stauber-Bagnall discloses: detecting, via the one or more input devices, an undo input directed to an undo selectable object; in response to detecting the undo input directed to the undo selectable object, undoing a first modification that is a most recently performed modification; subsequent to detecting the undo input, detecting, via the one or more input devices, a redo input directed to a redo selectable object; and in response to detecting the redo input, redoing the first modification (Stauber: [0123], a redo/undo queue of previous actions taken by the user).
As for claim 24, Stauber-Bagnall discloses: prior detecting the request to modify the media item, displaying, via the one or more display generation components t, a first media editing user interface that includes a first selectable object that corresponds to a first category of one or more editing tools; detecting, via one or more input devices, an input directed to the first selectable object; and
in response to detecting the input directed to the first selectable object, displaying, via the one or more display generation components, a second media editing user interface that includes a first set of editing selectable objects that, when selected, modifies the media item via one or more machine learning algorithms, wherein the second media editing user interface is displayed when the request to modify the media item is detected, and wherein at least the portion of first background element is modified via the one or more machine learning algorithms (Bagnall: [0003], a system utilizes one or more machine learning models to learn/identify characteristics of a digital image, anticipate potential edits to the digital image, and/or generate supplementary components that are usable in various edits; [0150], the scene-based image editing system 106 utilizes a machine learning model, such as a content-aware hole-filling machine learning model to generate the content fill(s) for each foreground object).
As for claim 25, Stauber-Bagnall discloses: a second selectable object, different from the first selectable object, that corresponds to a second category of one or more editing tools, wherein the second selectable object is concurrently displayed with the first selectable object; the one or more programs further include instructions for: after detecting the input directed to the first selectable object, detecting, via one or more input devices, an input directed to the second selectable object; and in response to detecting the input directed to the second selectable object: displaying, via the one or more generation components, the second category of one or more editing tools without displaying first category of one or more editing tools, wherein the second category of one or more editing tools includes: an editable selectable object for adjusting a first editable parameter of the media item; and an editable selectable object for adjusting a second editable parameter of the media item that is different from the first editable parameter (Bagnall: [0248], the image analysis graph 1000 associates characteristic attributes with one or more of the nodes 1004a-1004g to represent characteristic attributes of the corresponding characteristic categories).
As for claim 26, Stauber-Bagnall discloses: wherein the second media editing user interface includes an indication of one or more input criteria that needs to be satisfied in order to modify the media item via the one or more machine learning algorithms (Stauber: [0194], the animation progresses in accordance with an intensity of contact 562, as shown in FIGS. 5F-5G, where the representations 578A-578C appear and move upwards as the intensity of contact 562 increases toward the deep press intensity threshold (e.g., “IT.sub.D”). In some embodiments, the intensity, on which the progress of the animation is based, is the characteristic intensity of the contact).
As per claim 27, it recites features that are substantially same as those features claimed by claim 1, thus the rationales for rejecting claim 1 are incorporated herein.
As per claim 28, it recites features that are substantially same as those features claimed by claim 1, thus the rationales for rejecting claim 1 are incorporated herein.
Examiner’s Note
Examiner has cited particular columns/paragraph and line numbers in the references applied to the claims above for the convenience of the applicant. Although the specified citations are representative of the teachings of the art and are applied to specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested from the applicant in preparing responses, to fully consider the references in entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the Examiner.
In the case of amending the Claimed invention, Applicant is respectfully requested to indicate the portion(s) of the specification which dictate(s) the structure relied on for proper interpretation and also to verify and ascertain the metes and bounds of the claimed invention. This will assist in expediting compact prosecution. MPEP 714.02 recites: “Applicant should also specifically point out the support for any amendments made to the disclosure. See MPEP § 2163.06. An amendment which does not comply with the provisions of 37 CFR 1.121(b), (c), (d), and (h) may be held not fully responsive. See MPEP § 714.” Amendments not pointing to specific support in the disclosure may be deemed as not complying with provisions of 37 C.F.R. 1.131(b), (c), (d), and (h) and therefore held not fully responsive. Generic statements such as “Applicants believe no new matter has been introduced” may be deemed insufficient.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure. Applicants are required under 37 C.F.R. § 1.111(c) to consider these references fully when responding to this action.
Brandt (US Publication 20240169624) MODIFYING DIGITAL IMAGES VIA SCENE-BASED EDITING USING IMAGE UNDERSTANDING FACILITATED BY ARTIFICIAL INTELLIGENCE
Manzari (US Publication 20220319100) USER INTERFACES SIMULATED DEPTH EFFECTS
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 U.S.P.Q. 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 U.S.P.Q. 275, 277 (C.C.P.A. 1968)).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Hua Lu whose telephone number is 571-270-1410 and fax number is 571-270-2410. The examiner can normally be reached on Mon-Fri 9:00 am to 6:00 pm EST. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Scott Baderman can be reached on 571-272-3644. The fax phone number for the organization where this application or proceeding is assigned is 703-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Hua Lu/
Primary Examiner, Art Unit 2118
Prosecution Insights