DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1 – 20 are rejected under 35 U.S.C. 103 as being unpatentable over Cohen (Publication: US 2019/0196698 A1) in view of Aberman et al. (Publication: US 2023/0015117 A1 ).
Regarding claim 1, see rejection on claim 9.
Regarding claim 2, see rejection on claim 10.
Regarding claim 3, see rejection on claim 11.
Regarding claim 4, see rejection on claim 12.
Regarding claim 5, see rejection on claim 13.
Regarding claim 6, see rejection on claim 14.
Regarding claim 7, see rejection on claim 15.
Regarding claim 8, see rejection on claim 16.
Regarding claim 9, Cohen discloses a medium capable of being executed by a computer processor, the computer program instructions defining steps of ([0181] – Fig. 8, a system has a medium that stores program and executed by a processor to perform the following) :
identifying, by a processor, an image ([0005], [0117], [0182] - a vision module specific to the object is used, such as using a sky vision module including a neural network trained to identify skies when satisfying the replace request “Replace the boring sky with a cloudy sky”, executed by a processor.);
receiving, by the processor, natural language instructions for editing the image, the natural language instructions including a location within the image and an editing instruction ([0005], [0117], [0182] - a vision module specific to the object is used, such as using a sky vision module including a neural network trained to identify skies when satisfying the replace request “Replace the boring sky with a cloudy sky”, “natural language instructions”, instructions received by a processor .
[0093] to [0095], As showing Fig. 2, the location of the sky is identified and then replace the sky with different scene.
PNG
media_image1.png
534
768
media_image1.png
Greyscale
);
editing, by a machine learning model executed by the processor, the location within the image based on the natural language instructions by (
[0005], [0117] - a vision module specific to the object is used, such as using a sky vision module including a neural network trained to identify skies when satisfying the replace request “Replace the boring sky with a cloudy sky”.
[0093] to [0095], As showing Fig. 2, the location of the sky is identified based on the text instruction and then replace the sky with different scene, “edit”.
PNG
media_image1.png
534
768
media_image1.png
Greyscale
):
identifying a region within the image that corresponds to the location in the natural language instructions (
[0005], [0117] - a vision module specific to the object is used, such as using a sky vision module including a neural network trained to identify skies when satisfying the replace request “Replace the boring sky with a cloudy sky”, “location”.
[0093] to [0095], As showing Fig. 2, the location of the sky is identified based on the text instruction and then replace the sky with different scene, “edit”.
PNG
media_image1.png
534
768
media_image1.png
Greyscale
); and
editing the identified region by applying the editing instruction to the identified region to generate an edited image (
[0005], [0117] - a vision module specific to the object is used, such as using a sky vision module including a neural network trained to identify skies when satisfying the replace request “Replace the boring sky with a cloudy sky”, “identified sky region”.
[0093] to [0095], As showing Fig. 2, the location of the sky is identified based on the text instruction and then replace the sky with different scene, “edit”.
PNG
media_image1.png
534
768
media_image1.png
Greyscale
); and
causing, by the processor, display of the edited image (
[0093] to [0095], As showing Fig. 2, display 206 is edit image, the location of the sky is identified based on the text instruction and then replace the sky with different scene, “edit”.
PNG
media_image1.png
534
768
media_image1.png
Greyscale
).
Cohen does not disclose; however Aberman discloses
a non-transitory computer-readable storage medium for tangibly storing computer program instructions ([0064] - the model trainer 160 includes one or more sets of computer-executable instructions that are stored in a tangible computer-readable storage medium such as RAM hard disk or optical or magnetic media.)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Cohen with a non-transitory computer-readable storage medium for tangibly storing computer program instructions as taught by Aberman. The motivation for doing is to have the implementations to be executed in different platforms thus enhance configuration flexibility [064], [0108] .
Regarding claim 10, Cohen in view of Aberman disclose all the limitations of claim 9.
Cohen discloses wherein identifying the region within the image that corresponds to the location in the natural language instructions comprises identifying a landmark location within the image that is described by the natural language instructions (
[0005], [0117] - a vision module specific to the object is used, such as using a sky vision module including a neural network trained to identify skies when satisfying the replace request “Replace the boring sky with a cloudy sky”.
[0093] to [0095], As showing Fig. 2, the location of the sky is identified and then replace the sky with different scene.
PNG
media_image1.png
534
768
media_image1.png
Greyscale
).
Regarding claim 11, Cohen in view of Aberman disclose all the limitations of claim 9.
Cohen discloses identifying a set of objects of a similar type depicted within the image ([0005], [0117] - a vision module specific to the object is used, such as using a sky vision module including a neural network trained to identify skies when satisfying the replace request “Replace the boring sky with a cloudy sky”.
[0093] to [0095], As showing Fig. 2, the location of the sky is identified and then replace the sky with different scene.
PNG
media_image1.png
534
768
media_image1.png
Greyscale
); and
identifying, based on the natural language instructions, a specific object within the set of objects referred to by the natural language instructions ( [0005], [0117] - a vision module specific to the object is used, such as using a sky vision module including a neural network trained to identify skies when satisfying the replace request “Replace the boring sky with a cloudy sky”.
[0093] to [0095], As showing Fig. 2, the location of the sky is identified and then replace the sky with different scene.
PNG
media_image1.png
534
768
media_image1.png
Greyscale
).
Regarding claim 12, Cohen in view of Aberman disclose all the limitations of claim 9.
Cohen discloses identifying a type of object described by the natural language instructions ( [0005], [0117] - a vision module specific to the object is used, such as using a sky vision module including a neural network trained to identify skies when satisfying the replace request “Replace the boring sky with a cloudy sky”, “type”.
[0093] to [0095], As showing Fig. 2, the location of the sky is identified and then replace the sky with different scene.
PNG
media_image1.png
534
768
media_image1.png
Greyscale
);
and locating an object of the type within the image ( [0005], [0117] - a vision module specific to the object is used, such as using a sky vision module including a neural network trained to identify skies when satisfying the replace request “Replace the boring sky with a cloudy sky”.
[0093] to [0095], As showing Fig. 2, the location of the sky is identified, located, and then replace the sky with different scene.
PNG
media_image1.png
534
768
media_image1.png
Greyscale
).
Regarding claim 13, Cohen in view of Aberman disclose all the limitations of claim 9.
Cohen discloses identifying a relative directional descriptor within the natural language instructions (
[0119] - vision module 146 has semantic understanding of an image to be edited, and thus is able to distinguish between similar objects in an image. For instance, for an image containing a plurality of bookshelves, vision module 146 is able to distinguish a corner bookshelf from an object description indicating “the bookshelf in the corner”, “direction”.
[0005], [0117] - a vision module specific to the object is used, such as using a sky vision module including a neural network trained to identify skies when satisfying the replace request “Replace the boring sky with a cloudy sky”.
[0093] to [0095], As showing Fig. 2, the location of the sky is identified and then replace the sky with different scene.
PNG
media_image1.png
534
768
media_image1.png
Greyscale
); and
identifying the region at least in part based on the relative directional descriptor ( [0005], [0117] - a vision module specific to the object is used, such as using a sky vision module including a neural network trained to identify skies when satisfying the replace request “Replace the boring sky with a cloudy sky”.
[0093] to [0095], As showing Fig. 2, the location of the sky is identified and then replace the sky with different scene.
PNG
media_image1.png
534
768
media_image1.png
Greyscale
).
Regarding claim 14, Cohen in view of Aberman disclose all the limitations of claim 9.
Cohen discloses identifying a set of triplets that each comprise ([0005], [0117] - a vision module specific to the object is used, such as using a sky vision module including a neural network trained to identify skies when satisfying the replace request “Replace the boring sky with a cloudy sky”.
202 is an unmodified training image. Description of “Replace the boring sky with a cloudy sky”, and 206 is a modified training image. “triplets”
[0093] to [0095], As showing Fig. 2, the location of the sky is identified and then replace the sky with different scene.
PNG
media_image1.png
534
768
media_image1.png
Greyscale
):
an unmodified version of a training image ([0005], [0117] - a vision module specific to the object is used, such as using a sky vision module including a neural network trained to identify skies when satisfying the replace request “Replace the boring sky with a cloudy sky”.
202 is an unmodified training image.
[0093] to [0095], As showing Fig. 2, the location of the sky is identified and then replace the sky with different scene.
PNG
media_image1.png
534
768
media_image1.png
Greyscale
);
text that comprises a description of a location within the unmodified version of the training image ([0005], [0117] - a vision module specific to the object is used, such as using a sky vision module including a neural network trained to identify skies when satisfying the replace request “Replace the boring sky with a cloudy sky”, “text”.
[0093] to [0095], As showing Fig. 2, the location of the sky is identified and then replace the sky with different scene.
PNG
media_image1.png
534
768
media_image1.png
Greyscale
); and
a modified version of the training image that comprises a modification to the location described within the text ([0005], [0117] - a vision module specific to the object is used, such as using a sky vision module including a neural network trained to identify skies when satisfying the replace request “Replace the boring sky with a cloudy sky”.
206 is a modified of the training image.
[0093] to [0095], As showing Fig. 2, the location of the sky is identified and then replace the sky with different scene.
PNG
media_image1.png
534
768
media_image1.png
Greyscale
).
Aberman discloses providing the data to the machine learning model as input data ([0037] The systems and methods of the present disclosure provide several technical effects and benefits. As one example, the machine learning system can aid in computing performance by refining parameters of the image editing operator for processing the raw image data into processed image data. Thus, the performed image editing can be higher quality (e.g., more accurate) than previous techniques, which represents an improvement in the performance of a computing system.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Cohen in view of Aberman with providing the set to the machine learning model as input data as taught by Aberman. The motivation for doing is to have the implementations to be executed in different platforms thus enhance configuration flexibility [064], [0108] .
Regarding claim 15, Cohen in view of Aberman disclose all the limitations of claim 9.
Cohen discloses wherein the image comprises a frame of a video.([0039] – image storage service includes videos.)
Regarding claim 16, Cohen in view of Aberman disclose all the limitations of claim 15.
Cohen discloses the location within the image based on the natural language instructions comprises editing consecutive frames of the video (
[0039] – image storage service includes videos.
[0005], [0117] - a vision module specific to the object is used, such as using a sky vision module including a neural network trained to identify skies when satisfying the replace request “Replace the boring sky with a cloudy sky”.
[0093] to [0095], As showing Fig. 2, the location of the sky is identified and then replace the sky with different scene.
PNG
media_image1.png
534
768
media_image1.png
Greyscale
).
Regarding claim 17, see rejection on claim 9.
Regarding claim 18, see rejection on claim 10.
Regarding claim 19, see rejection on claim 11.
Regarding claim 20, see rejection on claim 12.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MING WU whose telephone number is (571)270-0724. The examiner can normally be reached on Monday - Thursday and alternate Fridays: 9:30am - 6:00pm EST .
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached on 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MING WU/
Primary Examiner, Art Unit 2618
Prosecution Insights