DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 09/09/2024 and 10/06/2025 is/are compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Office Action Summary
Claim(s) 1-4, 9-14, and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kansara (US 2022/0207645 A1) in view of Liu (US 2023/0091549 A1).
Claim(s) 5 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kansara (US 2022/0207645 A1) in view of Liu (US 2023/0091549 A1), further in view of Zhou et al (US 2024/0331214 A1).
Claim(s) 6-8 and 16-18 is/are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-4, 9-14, and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kansara (US 2022/0207645 A1) in view of Liu (US 2023/0091549 A1).
Regarding claim(s) 1 and 11, Kansara teaches a system comprising:
one or more processors (Figure 8; and Paragraph [0070]: “computing device 800 includes, without limitation, a processor 810, input/output (I/O) devices 820, and a memory 830”); and
one or more non-transitory computer-readable storage devices storing computing instructions that, when executed on the one or more processors (Figure 8; and Paragraph [0070]: “computing device 800 includes, without limitation, a processor 810, input/output (I/O) devices 820, and a memory 830”), cause the one or more processors to perform functions comprising:
receiving a source image corresponding to an electronic advertisement (Paragraph [0007]: “The technique includes computing a first visual interest score for a first visual interest region within a digital image based on content included in the first visual interest region […] technique further includes setting a location within the first visual interest region as a point of visual interest and transmitting the digital image […]”);
receiving a plurality of target display specifications (Paragraph [0003] – Paragraph [0006]: “[…] visual content is now commonly viewed on display devices having different display aspect ratios than the original format of the image or video […]”; and Paragraph [0007]: “dynamically cropping image data transmitted to an endpoint device”);
generating, using an image adaptation network, a target image set for the electronic advertisement that comprises target images compliant with each of the target display specifications (Paragraph [0003] – Paragraph [0006]; Paragraph [0007]; and Paragraph [0034]: “Files 218 include a plurality of digital visual content items, such as videos and still images. In addition, files 218 may include dynamic cropping metadata […] for a particular digital visual content item enables endpoint device 115 to dynamically crop the digital visual content item”), wherein generating the target image set comprises:
storing the target image set to enable the electronic advertisement to be displayed according to each of the plurality of target display specifications (Paragraph [0003] – Paragraph [0006]; Paragraph [0007]; and Paragraph [0034]: “dynamic cropping metadata associated with digital visual content items may instead be stored in a control server 120, or in any other technically feasible location”).
Kansara fails to teach to analyzing, using a saliency model of the image adaptation network, the source image to detect a salient feature region in the source image; and generating the target images for the target image set based, at least in part, on the salient feature region detected in the source image
However, Liu teaches to analyzing, using a saliency model of the image adaptation network, the source image to detect a salient feature region in the source image (Paragraph [0007]: “[…] performing target object detection on an initial image to obtain an object detection result, and performing image saliency detection on the initial image to obtain a saliency detection result; cropping the initial image based on the object detection result and the saliency detection result to obtain a corresponding cropped image”; and Paragraph [0034]: “Saliency detection: extraction of salient regions (i.e., regions of human interest) in an image by simulating human visual characteristics through an intelligent algorithm”); and
generating the target images for the target image set based, at least in part, on the salient feature region detected in the source image (Paragraph [0007]: “[…] performing target object detection on an initial image to obtain an object detection result, and performing image saliency detection on the initial image to obtain a saliency detection result; cropping the initial image based on the object detection result and the saliency detection result to obtain a corresponding cropped image”; and Paragraph [0049]: “Saliency detection: extraction of salient regions (i.e., regions of human interest) in an image by simulating human visual characteristics through an intelligent algorithm”).
Kansara teaches dynamically generating images compliant with different display aspect ratios by determining and transmitting a point of visual interest for cropping digital images. Liu teaches analyzing an image using a saliency model to detect a salient feature region and cropping the image based on the saliency detection result to generate a target image.
Therefore, it would have been obvious to one of ordinary skill in the art to combine Liu’s saliency based image analysis with Kansara’s display specific image adaptation in order to generate a set of target images corresponding to an electronic advertisement, each target image being compliant with a respective display specification while preserving salient visual features, as this represents a predictable use of known image processing techniques to improve visual presentation across different display environments. This motivation for the combination of Kansara and Liu is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim(s) 2 and 12, Kansara as modified by Liu teaches the system of claim 1, where Kansara teaches wherein the method further comprises:
in response to receiving a request to display the electronic advertisement (Paragraph [0043]: “the playback application 436 is configured to request and receive content from the content server 105 via the network interface 418. Further, the playback application 436 is configured to interpret the content and present the content via display device 450 and/or user I/O devices 452”), identifying a target display specification according to which the electronic advertisement will be displayed (Paragraph [0044]: “The dynamic cropping metadata for a particular digital visual content item enables an endpoint device to dynamically crop that particular digital visual content item so that the most visually interesting or significant subject matter in the digital visual content item is displayed. Thus, even when the display aspect ratio of the digital visual content item varies significantly from the display aspect ratio of a display device […]”);
retrieving a target image from the target image set based on the target display specification (Paragraph [0044]: “The dynamic cropping metadata […] enables an endpoint device to dynamically crop […] so that […] the display aspect ratio of the digital visual content item varies significantly from the display aspect ratio of a display device […] hereinafter referred to as the point of visual interest, remains viewable”); and
transmitting the target image to a user computer for display (Paragraph [0007]: “The technique further includes setting a location within the first visual interest region as a point of visual interest and transmitting the digital image and the location of the point of visual interest to a computing device for displaying a portion of the digital image that includes the point of visual interest”).
Regarding claim(s) 3 and 13, Kansara as modified by Liu teaches the system of claim 1, where Liu teaches wherein:
the saliency model is trained to identify the salient feature region in the source image (Paragraph [0007]: “[…] performing target object detection on an initial image to obtain an object detection result, and performing image saliency detection on the initial image to obtain a saliency detection result; cropping the initial image based on the object detection result and the saliency detection result to obtain a corresponding cropped image”; and Paragraph [0034]: “Saliency detection: extraction of salient regions (i.e., regions of human interest) in an image by simulating human visual characteristics through an intelligent algorithm”);
the source image is received as an input to the saliency model (Paragraph [0007]: “[…] performing image saliency detection on the initial image to obtain a saliency detection result […]”; and Paragraph [0034]: “Saliency detection: extraction of salient regions (i.e., regions of human interest) in an image by simulating human visual characteristics through an intelligent algorithm”); and
the saliency model is configured to analyze the source image and generate an output (read as “saliency detection result”) identifying the salient feature region in the source image (Paragraph [0013]: “performing target object detection and a saliency detection result obtained by performing saliency detection on the initial image, so as to retain a face and a saliency feature in the initial image”).
Regarding claim(s) 4 and 14, Kansara as modified by Liu teaches the system of claim 3, where Liu teaches wherein:
a saliency resizing function receives the salient feature region output by the saliency model (Paragraph [0071] – Paragraph [0072]: “determining an image size corresponding to the cropped image based on the application scenario type; and cropping the initial image based on the object detection result and the saliency detection result to obtain a cropped image corresponding to the image size […]”) and where Kansara teaches a target display specification for a target image (Paragraph [0003] – Paragraph [0006]: “[…] visual content is now commonly viewed on display devices having different display aspect ratios than the original format of the image or video […]”; and Paragraph [0007]: “dynamically cropping image data transmitted to an endpoint device”); and
where Liu teaches the saliency resizing function crops the source image based on the target display specification in a manner that preserves the salient feature region in the source image (Paragraph [0007]: “[…] performing target object detection on an initial image to obtain an object detection result, and performing image saliency detection on the initial image to obtain a saliency detection result; cropping the initial image based on the object detection result and the saliency detection result to obtain a corresponding cropped image”; and Paragraph [0034]: “Saliency detection: extraction of salient regions (i.e., regions of human interest) in an image by simulating human visual characteristics through an intelligent algorithm”).
Regarding claim(s) 9 and 19, Kansara as modified by Liu teaches the system of claim 1, where Kansara teaches wherein the plurality of target display specifications define different aspect ratios or dimensions for outputting the target images across heterogenous display environments associated with an electronic platform (Paragraph [0003] – Paragraph [0006]: “[…] visual content is now commonly viewed on display devices having different display aspect ratios than the original format of the image or video […]”; and Paragraph [0040]: “Examples of suitable devices known in the art that can display video frames and generate an acoustic output include televisions, smartphones, smartwatches, electronic tablets, and the like.”).
Regarding claim(s) 10 and 20, Kansara as modified by Liu teaches the system of claim 1, wherein where Liu teaches the image adaptation network comprises one or more segmentation models that are configured to extract salient objects from the source image in generating one or more target images (Paragraph [0007]; Paragraph [0049]: “Saliency detection: extraction of salient regions (i.e., regions of human interest) in an image by simulating human visual characteristics through an intelligent algorithm”; and Paragraph [0052]: “after target object detection is performed in the initial image to obtain a target object, an obtained object detection result is used for indicating an object region of the target object in the initial image. The saliency detection result obtained by performing saliency detection on the initial image is used for indicating a saliency feature region of the saliency feature in the initial image[…]”).
Claim(s) 5 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kansara (US 2022/0207645 A1) in view of Liu (US 2023/0091549 A1), further in view of Zhou et al (US 2024/0331214 A1).
Regarding claim(s) 5 and 15, Kansara as modified by Liu teaches the system of claim 1, but do not specifically teach wherein the image adaptation network comprises an outpainting network that is adapted to generate pixel content for at least one target image included in the target image set.
However, Zhou teaches wherein the image adaptation network comprises an outpainting network that is adapted to generate pixel content for at least one target image included in the target image set (Paragraph [0029]: “In some cases, the target aspect ratio is different from the initial aspect ratio. Further, the image framing component positions the image within the image frame having the target aspect ratio. For example, the image frame includes an image region containing the image and one or more extended regions outside the boundaries of the image. Finally, a generative neural network is used to generate an extended image”; and Paragraph [0064]: “For example, a user may select a text-guided outpainting option for image extension. Accordingly, a user may specify (e.g., type, input, etc.) text for content filling of the image, and the user may provide the text prompt to the image processing system (e.g., such as image processing system 200 described with reference to FIG. 2)”).
It would have been obvious to one of ordinary skill in the art at the time of the invention to incorporate Zhou’s generative image extension technique into Kansara’s image adaptation system in order to generate supplemental pixel content when adapting images for different display specifications, particularly where cropping alone would result in loss of image content. Further, it would have been obvious to apply Liu’s saliency based region identification to such a system to ensure that salient features of the source image are preserved while generating additional image content beyond the boundaries of the source image. The combination of Kansara, Liu, and Zhou merely applies known saliency guided image analysis and generative image extension techniques to a known image adaptation framework to achieve the predictable result of adapting images for different display environments while generating supplemental pixel content outside the source image. This motivation for the combination of Kansara, Liu, and Zhou is/are supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Allowable Subject Matter
Claim(s) 6-8 and 16-18 is/are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Relevant Prior Art Directed to State of Art
Ku et al (US 2023/0206614 A1) are relevant prior art not applied in the rejection(s) above. Ku discloses a method comprising: retrieving, a first image including a salient object corresponding to a domain; extracting, by a salient object detection model, the salient object from the first image; providing, the salient object to a generative model; and generating, by the generative model, a second image comprising the salient object and a staged background.
Jing et al (US 2017/0213248 A1) are relevant prior art not applied in the rejection(s) above. Jing discloses a computer system for selecting a content item, the computer system comprising: one or more computer storage devices storing images, storing content items, and storing instructions that are executable by one or more processing devices; and the one or more processing devices to execute the instructions to perform operations comprising: receiving data representing a first image from a content server of a publisher and over a computer network, the first image being part of a media property of the publisher; analyzing the first image to identify, based on at least one attribute of the first image, a region of interest of the first image that relates to a subject of the image and that is visually-distinguishable from a background of the image; decomposing the region of interest of the first image into one or more local features, wherein a local feature comprises an element of the region of interest; comparing the first image and a second image stored in the one or more computer storage devices by comparing the one or more local features included in the region of interest of the first image with one or more local features included in a region of interest of the second image to generate a match score indicating a likelihood of a match between the first image and the second image; identifying a content item associated with the second image based on an association between the second image and the content item in an association data store; selecting, from the one or more computer storage devices, the identified content item based on a comparison between the match score and a predetermined threshold; and outputting over the computer network, data representing the content item to at least one of the content server of the publisher and a client device.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONGBONG NAH whose telephone number is (571) 272-1361. The examiner can normally be reached M - F: 9:00 AM - 5:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ONEAL MISTRY can be reached on 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JONGBONG NAH/Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674