DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 9 objected to because of the following informalities:
In Claim 9, line 11, “with regions;.” should read “with regions.”.
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
an ‘object detection component’ in Claims 10 and 16,
an ‘object inference component’ in Claims 10 and 16,
an ‘object augmentation component’ in Claim 10,
an ‘object region detection and re-ranking component’ in Claims 10 and 16, and
an ‘image cropping component’ in Claims 10 and 16.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 17-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 17-20 recite the limitation of “the computer system of claim 15”. There is insufficient antecedent basis for this limitation in the claims. There is no ‘computer system’ claimed in Claim 15; the ‘computer system’ is only introduced in Claim 16. The Examiner suggests amending Claims 17-20 to depend on Claim 16 to resolve the lack of antecedent basis.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-9 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because Claims 1-9 are directed towards software per se.
Under the broadest reasonable interpretation, this limitation includes products that do not have a physical or tangible form (see MPEP §2106.03, “Non-limiting examples of claims that are not directed to any of the statutory categories include: Products that do not have a physical or tangible form, such as information (often referred to as “data per se”) or a computer program per se (often referred to as “software per se”) when claimed as a product without any structural recitation... For example, the BRI of machine readable media can encompass non-statutory transitory forms of signal transmission, such as a propagating electrical or electromagnetic signal per se. See In re Nuijten, 500 F.3d 1346, 84 USPQ2d 1495 (Fed. Cir. 2007). When the BRI encompasses transitory forms of signal transmission, a rejection under 35 U.S.C. 101 as failing to claim statutory subject matter would be appropriate. Thus, a claim to a computer readable medium that can be a compact disc or a carrier wave covers a non-statutory embodiment and therefore should be rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter. See, e.g., Mentor Graphics v. EVE-USA, Inc., 851 F.3d at 1294-95, 112 USPQ2d at 1134 (claims to a "machine-readable medium" were non-statutory, because their scope encompassed both statutory random-access memory and non-statutory carrier waves)”).
The examiner suggests rewriting Claim 1 to read “A non-transitory computer storage media” to ensure that Claims 1-9 falls under the four statutory categories of invention.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1 and 2 are rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al. (US Pub No 20240312020), hereinafter Cheng in view of Kon (US Pub No 20090313141), hereinafter Kon.
As to Claim 1, Cheng teaches one or more computer storage media storing computer-useable instructions (see paragraph [0004], “a non-transitory computer-readable medium includes instructions that, when executed by a processor, cause the processor to control a system”, and see Fig. 9, main memory 906)
that, when used by one or more computing devices, cause the one or more computing devices to perform operations (see paragraph [0037], “FIG. 9 is a block diagram showing an example a computer system 900 upon which aspects of this disclosure may be implemented”), the operations comprising:
obtaining a source image (see paragraph [0025], “The source image may be received by the local device 110 and transmitted to the server 120”);
obtaining a set of guidelines for cropping the source image (see paragraph [0016], “User intention data 20 may be received, which indicates one or more target features desired by the user. The user intention data 20 may be any one or more of text data, audio data, image data, video data, etc.”, where the user intention data is interpreted as the guidelines for cropping the image);
generating a list of objects present in the source image (see paragraph [0025], “For example, the system 100 may perform context recognition by analyzing various features shown in the source image 10, which may be performed by the server 120 in corporation with the ML engine 130. The identified visual features of the source image 10 may include water, a river, a boat, a woman, three dogs, the boat floating on the river, the woman sitting on the boat.”
using an object recognition model (see paragraph [0022], “As an example, the ML engine 130 may be trained to identify a plurality of visual features in an image”);
generating a list of desired objects based, at least in part, on the set of guidelines for cropping the source image and obtaining object keywords (see paragraph [0016], “Contextual information (e.g., three dogs) may be extracted from the text data 20A”, where the ‘contextual information extracted’ is interpreted as the keywords);
identifying regions of the source image that contain at least one object of the list of objects (see paragraph [0033], “the system 100 (e.g., server 120/ML engine 130) may contextually compare the target feature 430 (e.g., three dogs) with each of the visual features 510 of the source image 10 to identify a portion or portions of the source image 10 that are contextually relevant to the target feature, which may result in identifying, for example, a portion 60A (shown in FIG. 6B) of the source image 10”),
and generating a cropped image from the source image (see paragraph [0034], “source image cropping 710 is performed to generate a set of cropped images 730”)
of a desired image size specified in the set of guidelines (see paragraph [0034], “Hence, when the source image cropping 710 is performed based on the cropping candidate portions 530, the system 100 may consider one or more cropping rules 720”, and see paragraph [0035], “The cropping rules 720 may also include various commonly used image configurations (e.g., image sizes, aspect ratios, image types, image compression ratios, data size limitations, etc.) which may be required by media/content creation industries, social networking platforms, etc.”),
wherein the cropped image at least includes a selected identified region of the identified regions (see paragraph [0028], “Based on the contextual relevance of each visual feature with respect to the target feature, at step 330, one or more cropping candidate portions may be identified from the source image.”).
Cheng fails to teach combining the list of objects and the list of desired objects to generate a list of object keywords and identifying regions of the source image that contain at least one object of the list of objects based, at least in part, on the list of object keywords. However, in the analogous field of image analysis, Kon teaches obtaining a list of objects in an image (see paragraph [0057], “For example, keywords that represent the contents of each image in a user's viewing history may be extracted, by obtaining tag information of the images”),
obtaining a list of objects desired objects (see paragraph [0033], “The term “user's history of activity” refers to keyword logs of searches input by the user into a personal computer”, where the Examiner has interpreted the keywords of searches input by the user to be the list of desired objects),
and combining the list of objects and the list of desired objects to generate a list of object keywords (see paragraph [0057], “Alternatively, keywords from keyword logs of searches input by the user may be counted, or the keywords extracted from the images and the keywords from the keyword logs of searches may be combined and counted”).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the keyword combining taught by Kon with the image-cropping system taught by Cheng. The motivation for doing so would be to use the keywords to produce rankings which can be used to identify regions desired by the user. Kon teaches in paragraph [0057], “Rankings of keywords such as that illustrated in FIG. 4 are produced based on the counts. It is estimated that objects having images thereon related to keywords having high rankings are desired by the user”. Thus, it would have been obvious to combine the keywords and ranking taught by Kon with the image-cropping system taught by Cheng in order to obtain the invention as claimed in Claim 1.
As to Claim 2, Cheng in view of Kon teaches the source image is one of a plurality of images (see Cheng, paragraph [0016], “Also, a large number of images may be searched to find those images containing the visual features desired by the user”).
and generating the list of objects comprises generating a list of objects that are present in at least one image of the plurality of images, (see Cheng, paragraph [0025], “ For example, the system 100 may perform context recognition by analyzing various features shown in the source image 10, which may be performed by the server 120 in corporation with the ML engine 130. The identified visual features of the source image 10 may include water, a river, a boat, a woman, three dogs, the boat floating on the river, the woman sitting on the boat”)
using the object recognition model (see paragraph [0022], “As an example, the ML engine 130 may be trained to identify a plurality of visual features in an image”).
Claims 3 is rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al. (US Pub No 20240312020), hereinafter Cheng in view of Kon (US Pub No 20090313141), hereinafter Kon, and further in view of Saa-Garcia et al. (US Pub No 20230036950), hereinafter Saa-Garcia.
As to Claim 3, Cheng fails to teach assigning a ranking to each of the object keywords of the list of object keywords; and wherein the selected identified region is selected based, at least in part, on the rankings of the object keywords of objects in the selected identified region.
However, in an analogous art, Kon teaches assigning a ranking to object keywords (see paragraph [0065], “Rankings of keywords such as that illustrated in FIG. 4 are produced based on the counts. It is estimated that objects having images thereon related to keywords having high rankings are desired by the user”). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the keyword combining taught by Kon with the image-cropping system taught by Cheng. The motivation for doing so would be to use the keywords to produce rankings which can be used to identify regions desired by the user. Kon teaches in paragraph [0057], “Rankings of keywords such as that illustrated in FIG. 4 are produced based on the counts. It is estimated that objects having images thereon related to keywords having high rankings are desired by the user”. Thus, it would have been obvious to combine the keywords and ranking taught by Kon with the image-cropping system taught by Cheng.
Both Cheng and Kon fail to teach that the selected identified region is selected based, at least in part, on the rankings of the object keywords of objects in the selected identified region. However, in an analogous art, Saa-Garcia teaches a method for cropping an image (see abstract) which comprises obtaining rankings for objects in an image (see paragraph [0065], “The next step is an optional step and ranks the detected objects in order of importance (step S804))”,
and selecting an identified region to crop based on the ranking of objects (see paragraph [0037], “For example, the method may further comprise ranking the plurality of detected objects and cropping the input image by centering on a highest ranked object”, where the highest ranked object is the selected region). Thus, it would have been obvious to one of the art before the effective filing date of the claimed invention to combine the region identification taught by Saa-Garcia with the keyword ranking taught by Kon. The motivation for doing so would be to use the cropped image contains important objects. Saa-Garcia teaches in paragraph [0037], “For example, cropping the input image may include cropping the image to center on one of the plurality of detected objects. The object on which to center the cropped image may be selected in different ways. For example, the method may further comprise ranking the plurality of detected objects and cropping the input image by centering on a highest ranked object.” Thus, it would have been obvious to combine the region identification by ranking taught by Saa-Garcia with the teachings of Cheng and Kon in order to obtain the invention as claimed in Claim 3.
Claim(s) 4, 5,6, and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al. (US Pub No 20240312020), hereinafter Cheng in view of Kon (US Pub No 20090313141), hereinafter Kon, and further in view of Maragheh et al., (R. Y. Maragheh et al., "LLM-TAKE: Theme-Aware Keyword Extraction Using Large Language Models," 2023 IEEE International Conference on Big Data (BigData), Sorrento, Italy, 2023, pp. 4318-4324), hereinafter Maragheh.
As to Claim 4, Cheng in view of Kon teaches that that the set of guidelines for cropping the source image comprises a content brief that indicates a type of desired content (see Cheng, paragraph [0016], “User intention data 20 may be received, which indicates one or more target features desired by the user. The user intention data 20 may be any one or more of text data, audio data, image data, video data, etc. For example, user intention data 20 may be text data 20A containing letters “three dogs,” which may be entered or sent by the user”, where the text data is the ‘content brief’);
and one or more desired image sizes(see Cheng, paragraph [0034], “Hence, when the source image cropping 710 is performed based on the cropping candidate portions 530, the system 100 may consider one or more cropping rules 720”, and see paragraph [0035], “The cropping rules 720 may also include various commonly used image configurations (e.g., image sizes, aspect ratios, image types, image compression ratios, data size limitations, etc.) which may be required by media/content creation industries, social networking platforms, etc.”).
Cheng in view of Kon fails to teach that the list of desired objects is generated using a large-language model (LLM) based, at least in part, on the content brief.
However, in an analogous art, Maragheh teaches a large-language model which analyzes a content brief (see pg. 4318, Abstract, “In this paper, we explore using Large Language Models (LLMs) in generating keywords for items that are inferred from the items’ textual metadata”, where the textual metadata is the content brief),
which can be used to obtain a list of desired keywords for objects(see pg. 4318, Section I., “In this paper, we propose a multi-stage framework which utilizes the power of the large language models to derive theme-aware keywords for items”.
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the large-language model taught by Maragheh with the image cropping method taught by Cheng in view of Kon. The motivation for doing so would be to improve the quality of keywords extracted. Maragheh teaches on pg. 4319, Section I., “We discuss how each stage of the framework helps improve the quality of the output keywords and reduce hallucinations.” Thus, it would have been obvious to combine the LLM taught by Maragheh with the image-cropping method taught by Cheng and Kon in order to obtain the invention as claimed in Claim 4.
As to Claim 5, Cheng in view of Kon fails to teach the operations further comprise: augmenting the list of object keywords by removing one or more object keywords from the list of object keywords.
However, Maragheh teaches a list of object keywords can be augmented by removing one removing one or more object keywords from the list of object keywords (see pg. 4320, Section III, Subsection B., “The objective of the framework is to generate themes that are informative enough and can help the users in their decision journey. Thus, a very general theme may not have the differentiating power to help in user’s decision making process. Because of this we eliminate the very general words like “Perfect” or “Great” which do not add any information value to compare and contrast the product”).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the keyword augmentation taught by Maragheh with the image cropping method taught by Cheng in view of Kon. The motivation for doing so would be to improve the quality of keywords by removing very general words, as taught by Maragheh on pg. 4320. Thus, it would have been obvious to combine the keyword augmentation taught Maragheh with the teachings of Cheng and Kon in order to obtain the invention as claimed in Claim 5.
As to Claim 6, Chen in view of Kon fails to teach that the operations further comprise: augmenting the list of object keywords by adding one or more object keywords to the list of object keywords.
However, Maragheh teaches that a list of object keywords can be augmented by adding one or more object keywords to the list of object keywords (see pg. 4320, Section II., Subjection B., “The initial set of general themes is obtained from Top 500 adjectives used in Oxford [32]. More keywords are added to this set of general words”). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the keyword augmentation taught by Maragheh with the image cropping method taught by Cheng in view of Kon. The motivation for doing so would be to include keywords that are useful to the user. Maragheh teaches on pg. 4320, Section II., Subjection B., “The objective of the framework is to generate themes that are informative enough and can help the users in their decision journey.” Thus, it would have been obvious to combine the keyword augmentation taught Maragheh with the teachings of Cheng and Kon in order to obtain the invention as claimed in Claim 6.
As to Claim 8, Cheng in view of Kon fails to teach fails to teach combining the list of objects and the list of desired objects to generate a list of object keywords uses a large language model (LLM).
However, Maragheh teaches that sets of object keywords can be combined to generate a final list of object keywords by using the outputs of large language models (see Fig. 5, shown below, where two sets of keywords are combined using the LLM to generate a final set of keywords).
PNG
media_image1.png
779
1381
media_image1.png
Greyscale
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the LLM framework taught by Maragheh with the image cropping method taught by Cheng in view Kon. The motivation for doing so would be to improve the quality of keywords extracted (Maragheh teaches on pg. 4319, Section I.). Thus, it would have been obvious to combine the LLM framework taught by Maragheh with the teachings of Cheng and Kon in order to obtain the invention as claimed in Claim 8.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al. (US Pub No 20240312020), hereinafter Cheng in view of Kon (US Pub No 20090313141), hereinafter Kon, and further in view of Goyal et al. (US Pub No 12033348), hereinafter Goyal.
As to Claim 7, Cheng in view of Kon fails to teach wherein the source image is a frame of a video comprising a plurality of frames.
However, in an analogous art, Goyal teaches a method for generating cropped image (see abstract),
wherein the wherein the source image is a frame of a video comprising a plurality of frames (see Col. 1, lines 22-36, “The memory stores instructions to cause the processor to receive a video stream including a series of video frames…The memory also stores instructions to cause the processor to generate at least one image that depicts the object and that includes a cropped portion of a video frame from the series of video frames”).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the image-cropping method taught by Cheng in view of Kon in order to crop video data as taught by Goyal. The motivation for doing so would be to obtain cropped images of objects from video surveillance data. Goyal teaches in Col. 1, lines 13-17, “Object detection can include the detection of depicted objects such as people and license plates. Applications of object detection include, for example, video surveillance”. Thus, it would have been obvious to combine the teachings of Goyal with the teachings of Cheng and Kon in order to obtain the invention as claimed in Claim 7.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al. (US Pub No 20240312020), hereinafter Cheng in view of Kon (US Pub No 20090313141), hereinafter Kon, and further in view of Saa-Garcia et al. (US Pub No 20230036950), hereinafter Saa-Garcia, and further in view of Ren et al., (Ren, Tianhe, et al. “Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks.” ArXiv.org, 25 Jan. 2024”), hereinafter Ren.
As to Claim 9, Cheng in view of Kon fails to teach identifying regions of the source image that contain at least one object of the list of objects comprises: segmenting the source image using a segment anything model (SAM) to generate a list of identified regions; assigning a ranking to the identified regions of the list of identified regions; combining the list of identified regions and the list of object keywords to generate a list of objects with regions; sorting the list of objects with regions based, at least in part, on the ranking of the identified regions; and identifying the regions of the source image that contain at least one object of the list of objects based, at least in part, on the sorted list of objects with regions.
However, in an analogous art, Saa-Garcia teaches segmenting the source image (see paragraph [0064], “The next step is to detect the objects within the input image… pixel wise classification or other object detection methods may be used. Examples of suitable object detection algorithms include “Yolov3: An Incremental Improvement” by Redmon et al, “YOLACT++ Better Real-time Instance Segmentation” by Bolya et al and “Mask R-CNN” by He et al.”),
to generate a list of identified regions (see paragraph [0056], “In other words, the caption may describe a plurality of objects, e.g. all the objects in a part of the image”)
assigning a ranking to the identified regions of the list of identified regions (see paragraph [0065], “The next step is an optional step and ranks the detected objects in order of importance (step S804)”);
sorting the list of objects with regions based, at least in part, on the ranking of the identified regions and identifying the regions of the source image that contain at least one object of the list of objects based, at least in part, on the sorted list of objects with regions objects (see paragraph [0037], “For example, the method may further comprise ranking the plurality of detected objects and cropping the input image by centering on a highest ranked object”, where the highest ranked object is the selected region).
Saa-Garcia fails to teach combining the list of identified regions and the list of object keywords to generate a list of objects with region. However, Kon teaches that a list of keywords corresponding to an image and a list of user-desire object keywords can be combined (see Kon, paragraph [0057], “Alternatively, keywords from keyword logs of searches input by the user may be counted, or the keywords extracted from the images and the keywords from the keyword logs of searches may be combined and counted”).
Thus, it would have been obvious to one of the art before the effective filing date of the claimed invention to combine the region segmentation and ranking taught by Saa-Garcia with the keyword ranking taught by Kon in order to obtain the sorted region list. The motivation for doing so would be to ensure the final cropped image contains objects of interest to the user. Saa-Garcia teaches in paragraph [0037], “For example, cropping the input image may include cropping the image to center on one of the plurality of detected objects. The object on which to center the cropped image may be selected in different ways. For example, the method may further comprise ranking the plurality of detected objects and cropping the input image by centering on a highest ranked object.” Thus, it would have been obvious to combine the region identification by ranking taught by Saa-Garcia with the teachings of Cheng and Kon.
Cheng, Kon, and Saa-Garcia all fail to teach that the image segmentation is done by a segment anything model (SAM). However, in an analogous art, Ren et al. teaches using a Grounded SAM system (see pg. 1, Abstract, “We introduce Grounded SAM, which uses Grounding DINO [38] as an open-set object detector to combine with the segment anything model (SAM)”),
to segment an image and generate a list of identified regions (see pgs. 3-4, Section 3.2., “Given an input image and a text prompt, we first employ Grounding DINO to generate precise boxes for objects or regions within the image by leveraging the textual information as condition. Subsequently, the annotated boxes obtained through Grounding DINO serve as the box prompts for SAM to generate precise mask annotations”). Thus, it would have been obvious to one of ordinary skill in the art to substitute the segmentation model taught by Saa-Garcia with the SAM model taught by Ren. The motivation for doing so would be to use this model to handle intricate segmentation tasks. Ren teaches on pg. 2, Section 1, “Although there have been advances in addressing open world tasks through these methodologies, a robust pipeline capable of supporting complex and fundamental open-world tasks such as open-set segmentation is still lacking in the market… Grounded SAM offers a powerful and comprehensive platform that further facilitates an efficient fusion of different expert models to tackle more intricate open-world tasks.” Thus, it would have been obvious to combine the Grounded SAM model taught by Ren with the teachings of Cheng, Kon, and Saa-Garcia in order to obtain the invention as claimed in Claim 9.
Claim(s) 10, 11, 13-17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al. (US Pub No 20240312020), hereinafter Cheng in view of Kon (US Pub No 20090313141), hereinafter Kon, and further in view of Maragheh et al., (R. Y. Maragheh et al., "LLM-TAKE: Theme-Aware Keyword Extraction Using Large Language Models," 2023 IEEE International Conference on Big Data (BigData), Sorrento, Italy, 2023, pp. 4318-4324), hereinafter Maragheh, and further in view of Saa-Garcia (US Pub No 20230036950), hereinafter Saa-Garcia.
As to Claim 10, Cheng in view of Kon teaches a computer-implemented method comprising (see Cheng, paragraph [0003], “a method of cropping an image includes”):
generating, by an object detection component (see Chen, Fig. 1, ML engine 130),
a list of objects present in a digital asset (see Cheng, paragraph [0025], “ For example, the system 100 may perform context recognition by analyzing various features shown in the source image 10, which may be performed by the server 120 in corporation with the ML engine 130. The identified visual features of the source image 10 may include water, a river, a boat, a woman, three dogs, the boat floating on the river”, where the Examiner has interpreted the source image as the digital asset),
selected from a set of digital assets (see Cheng, paragraph [0016], “Also, a large number of images may be searched to find those images containing the visual features desired by the user”, where the Examiner has interpreted the large number of images to be the set of images, and the found images as the selected digital asset),
generating, by an object inference component (see Chen, Fig. 1, ML engine 130),
a list of desired objects and keywords based, at least in part, on a set of guidelines for cropping digital assets (see Cheng, paragraph [0016], “Contextual information (e.g., three dogs) may be extracted from the text data 20A”, where the ‘contextual information extracted’ is interpreted as the keywords, and the text data is interpreted as a guideline for cropping the digital asset);
combining the list of objects and the list of desired objects to generate a list of object keywords (see Kon, paragraph [0057], “Alternatively, keywords from keyword logs of searches input by the user may be counted, or the keywords extracted from the images and the keywords from the keyword logs of searches may be combined and counted”).
assigning a ranking to each object keyword in the list of object keywords to generate a ranked list of object keywords (see paragraph [0065], “Rankings of keywords such as that illustrated in FIG. 4 are produced based on the counts. It is estimated that objects having images thereon related to keywords having high rankings are desired by the user”)
identifying, by an object region detection component (see Chen, Fig. 1, ML engine 130),
regions of the selected digital asset that contain at least one object of the list of objects based, at least in part, on the list of object keywords (see Cheng, paragraph [0033], “the system 100 (e.g., server 120/ML engine 130) may contextually compare the target feature 430 (e.g., three dogs) with each of the visual features 510 of the source image 10 to identify a portion or portions of the source image 10 that are contextually relevant to the target feature, which may result in identifying, for example, a portion 60A (shown in FIG. 6B) of the source image 10”);
and generating, by an image cropping component, a cropped version of the selected digital asset that at least includes a selected identified region of the identified regions (see Cheng, paragraph [0034], “source image cropping 710 is performed to generate a set of cropped images 730”).
Cheng in view of Kon fails to teach combining, by the object inference component, the list of objects and the list of desired objects to generate a list of object keywords. Cheng in view of Kon fails to additionally teach assigning, by the object inference component, a ranking to each object keyword in the list of object keywords to generate a ranked list of object keywords, and augmenting, by an object augmentation component, the ranked list of object keywords by removing object keywords from the ranked list of object keywords based, at least in part, on the ranking of the object keywords.
However, in an analogous art, Maragheh teaches combining, by an object inference component (see Maragheh Fig. 5, LLM or “large language model”),
sets of object keywords in order to obtain a final keyword set (see Fig. 5, shown below, where two sets of keywords are combined using the LLM to generate a final set of keywords),
and assigning, by an object inference component, a ranking to each object keyword in the list of object keywords to generate a ranked list of object keywords (see pg. 4320, Section III., Subsection C., “For doing this, we perform another round of prompts asking the LLM to output a confidence score that measures how descriptive the generated keywords are for the input product., where the Examiner has interpreted the LLM to be the object inference component, and see pg. 4321, Section III., Subsection D., “We use the generated scores from the previous step, as the primary criteria of ranking.)”
and augmenting, by an object augmentation component (see pg. 4320, Section III., Subsection C., “For doing this, we perform another round of prompts asking the LLM to output a confidence score”, where the Examiner has interpreted the LLM to be the object augmentation component)
the ranked list of object keywords by removing object keywords from the ranked list of object keywords based, at least in part, on the ranking of the object keywords (see Maragheh, pg., 4321, Section III., Subsection E., “Finally, in order to further improve the final set of keyword themes generated for the items, we perform a synonymity check to avoid extracting keywords that are semantically similar. For instance, words ’fun’ and ’funny’ may appear as a theme for a given product. By doing this step, we eliminate the lower-rank theme which is semantically similar to a higher rank.”
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the ranking and augmenting of the object list taught by Mishra with the teachings of Cheng and Kon. The motivation for doing so would be to delete keywords that are synonyms for each other (see Maragheh pg., 4321, Section III., Subsection E.).
Cheng in view of Kon and Maragheh fails to teach identifying by an object region detection and re-ranking component, regions of the selected digital asset that contain at least one object of the list of objects based, at least in part, on the ranked list of object keywords.
However, in an analogous art, Saa-Garcia teaches an object region detection and re-ranking component (see paragraph [0081], “The artificial intelligence model may be obtained by training”, where the Examiner has interpreted the model to be the object region detection and re-ranking component),
that identifies regions of the selected digital asset that contain at least one object of the list of objects based, at least in part, on the ranked list (see paragraph [0037], “For example, the method may further comprise ranking the plurality of detected objects and cropping the input image by centering on a highest ranked object”, where the highest ranked object is the selected region).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the region identification taught by Saa-Garcia with the keyword ranking taught by Maragheh. The motivation for doing so would be to ensure the final cropped image contains objects of interest to the user. Saa-Garcia teaches in paragraph [0037], “For example, cropping the input image may include cropping the image to center on one of the plurality of detected objects. The object on which to center the cropped image may be selected in different ways. For example, the method may further comprise ranking the plurality of detected objects and cropping the input image by centering on a highest ranked object.” Thus, it would have been obvious to combine the teachings of Saa-Garcia with the teachings of Chen, Kon, and Maragheh in order to obtain the invention as claimed in Claim 10.
As to Claim 11, Cheng in view of Kon, Maragheh, and Saa-Garcia teaches the set of digital assets comprises one or more images (see Cheng, paragraph [0016], “Also, a large number of images may be searched to find those images containing the visual features desired by the user”).
As to Claim 13, Cheng in view of Kon, Maragheh, and Saa-Garcia teaches the cropped version of the selected digital asset is cropped based, at least in part, on an image size indicated by the set of guidelines for cropping digital assets and generating a cropped image from the source image (see Cheng, paragraph [0034], “Hence, when the source image cropping 710 is performed based on the cropping candidate portions 530, the system 100 may consider one or more cropping rules 720”, and see Cheng, paragraph [0035], “The cropping rules 720 may also include various commonly used image configurations (e.g., image sizes, aspect ratios, image types, image compression ratios, data size limitations, etc.) which may be required by media/content creation industries, social networking platforms, etc.”).
As to Claim 14, Cheng in view of Kon, Maragheh, and Saa-Garcia teaches that the cropped version of the selected digital asset is cropped based, at least in part, on an aspect ratio indicated by the set of guidelines for cropping digital assets (see Cheng, paragraph [0035], “The cropping rules 720 may also include various commonly used image configurations (e.g., image sizes, aspect ratios, image types, image compression ratios, data size limitations, etc.) which may be required by media/content creation industries, social networking platforms, etc.”).
As to Claim 15, Cheng teaches the list of objects comprises a list of objects that are present in at least one digital asset of the set of digital assets generating a list of objects present in a digital asset (see Cheng, paragraph [0025], “ For example, the system 100 may perform context recognition by analyzing various features shown in the source image 10”, where the Examiner has interpreted the source image as the digital and see Cheng, paragraph [0016], “Also, a large number of images may be searched to find those images containing the visual features desired by the user”, where the Examiner has interpreted the large number of images to be the set of images, and the found images as the selected digital asset).
Cheng fails to teach each object of the list of objects has an assigned ranking based, at least in part, on a number of occurrences of the object in the set of digital assets.
However, Kon teaches that ranking may be based on a number of occurrences (see paragraph [0057], “ Alternatively, keywords from keyword logs of searches input by the user may be counted, or the keywords extracted from the images and the keywords from the keyword logs of searches may be combined and counted. Rankings of keywords such as that illustrated in FIG. 4 are produced based on the counts”). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the ranking based on occurrences taught by Kon with the teachings of Cheng, Maragheh, and Saa-Garcia. The motivation for doing so would be to obtain cropped images that are of interest to the user. Kon teaches in paragraph [0057], “It is estimated that objects having images thereon related to keywords having high rankings are desired by the user”. Thus, it would have been obvious to combine the ranking taught by Kon with the teachings of Cheng, , Maragheh, and Saa-Garcia in order to obtain the invention as claimed in Claim 15.
As to Claim 16, Cheng in view of Kon teaches a computer system (see Cheng, Fig. 9, computer system 900) comprising:
one or more processors (see Cheng, paragraph [0037], “The computer system 900 may include a bus 902 or other communication mechanism for communicating information, and a processor 904 coupled with the bus 902 for processing information”, and see Fig. 9 processor 904);
and one or more computer storage media storing computer-useable instructions (see Cheng, paragraph [0004], “a non-transitory computer-readable medium includes instructions that, when executed by a processor, cause the processor to control a system”, and see Fig. 9, main memory 906)
generating, by an object detection component (see Cheng, Fig. 1, ML engine 130),
a list of objects present in at least one image (see Cheng, paragraph [0025], “For example, the system 100 may perform context recognition by analyzing various features shown in the source image 10, which may be performed by the server 120 in corporation with the ML engine 130. The identified visual features of the source image 10 may include water, a river, a boat, a woman, three dogs, the boat floating on the river”, where the Examiner has interpreted the source image as the digital asset),
of an image corpus (see Cheng, paragraph [0016], “Also, a large number of images may be searched to find those images containing the visual features desired by the user”)
generating, by an object inference component (see Cheng, Fig. 1, ML engine 130), a list of desired objects and keywords based, at least in part, on a set of guidelines for cropping digital assets (see Cheng, paragraph [0016], “Contextual information (e.g., three dogs) may be extracted from the text data 20A”, where the ‘contextual information extracted’ is interpreted as the keywords, and the text data is interpreted as a guideline for cropping the digital asset);
combining, by an object inference component the list of objects and the list of desired objects to generate a list of object keywords (see Kon, paragraph [0057], “Alternatively, keywords from keyword logs of searches input by the user may be counted, or the keywords extracted from the images and the keywords from the keyword logs of searches may be combined and counted”).
assigning a ranking to each object keyword in the list of object keywords to generate a ranked list of object keywords (see Kon paragraph [0065], “Rankings of keywords such as that illustrated in FIG. 4 are produced based on the counts. It is estimated that objects having images thereon related to keywords having high rankings are desired by the user”)
identifying, by an object region detection component (see Cheng, Fig. 1, ML engine 130), regions of the selected digital asset that contain at least one object of the list of objects based, at least in part, on the list of object keywords (see Cheng, paragraph [0033], “the system 100 (e.g., server 120/ML engine 130) may contextually compare the target feature 430 (e.g., three dogs) with each of the visual features 510 of the source image 10 to identify a portion or portions of the source image 10 that are contextually relevant to the target feature, which may result in identifying, for example, a portion 60A (shown in FIG. 6B) of the source image 10”);
and generating, by an image cropping component (see Cheng, Fig. 1, ML engine 130), a cropped version of the selected digital asset that at least includes a selected identified region of the identified regions (see Cheng, paragraph [0034], “source image cropping 710 is performed to generate a set of cropped images 730”).
Cheng in view of Kon fails to teach identifying regions of the selected digital asset that contain at least one object of the list of objects based, at least in part, on the ranked list of object keywords.
However, in an analogous art, Saa-Garcia teaches an object region detection and re-ranking component (see paragraph [0081], “The artificial intelligence model may be obtained by training”, where the Examiner has interpreted the model to be the object region detection and re-ranking component),
that identifies regions of the selected digital asset that contain at least one object of the list of objects based, at least in part, on the ranked list (see paragraph [0037], “For example, the method may further comprise ranking the plurality of detected objects and cropping the input image by centering on a highest ranked object”, where the highest ranked object is the selected region).
Thus, it would have been obvious to one of the art before the effective filing date of the claimed invention to combine the region identification taught by Saa-Garcia with the keyword ranking taught by Kon. The motivation for doing so would be to use the cropped image contains important objects. Saa-Garcia teaches in paragraph [0037], “For example, cropping the input image may include cropping the image to center on one of the plurality of detected objects. The object on which to center the cropped image may be selected in different ways. For example, the method may further comprise ranking the plurality of detected objects and cropping the input image by centering on a highest ranked object.”
Cheng in view of Kon and Saa-Garcia fails to teach an object inference component generates the list of object keywords and assigns rankings.
However, in an analogous art, However, in an analogous art, Maragheh teaches combining, by an object inference component (see Maragheh Fig. 5, LLM or “large language model”),
sets of object keywords in order to obtain a final keyword set (see Fig. 5, shown below, where two sets of keywords are combined using the LLM to generate a final set of keywords),
and assigning, by an object inference component, a ranking to each object keyword in the list of object keywords to generate a ranked list of object keywords (see pg. 4320, Section III., Subsection C., “For doing this, we perform another round of prompts asking the LLM to output a confidence score that measures how descriptive the generated keywords are for the input product., where the Examiner has interpreted the LLM to be the object inference component, and see pg. 4321, Section III., Subsection D., “We use the generated scores from the previous step, as the primary criteria of ranking.)”
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the LLM framework taught by Maragheh with the image cropping method taught by Cheng in view Kon. The motivation for doing so would be to improve the quality of keywords extracted (Maragheh teaches on pg. 4319, Section I.). Thus, it would have been obvious to combine the LLM framework taught by Maragheh with the teachings of Cheng and Kon in order to obtain the invention as claimed in Claim 16.
As to Claim 17, Cheng in view of Kon, Maragheh, and Saa-Garcia teaches augmenting, by an object augmentation component, the ranked list of object keywords by removing one or more object keywords from the list of object keywords based, at least in part, on the assigned ranking (see Maragheh, pg., 4321, Section III., Subsection E., “We use the generated scores from the previous step, as the primary criteria of ranking…we eliminate all of the item-theme pairs that have a score lower than a pre-specified threshold”).
As to Claim 19, Cheng in view of Kon and Saa-Garcia fails to explicitly teach that generating the list of object keywords uses a large language model (LLM).
However, in an analogous art, Maragheh teaches a large-language model which analyzes a content brief (see pg. 4318, Abstract, “In this paper, we explore using Large Language Models (LLMs) in generating keywords for items that are inferred from the items’ textual metadata”, where the textual metadata is the content brief),
which can be used to obtain a list of desired keywords for objects(see pg. 4318, Section I., “In this paper, we propose a multi-stage framework which utilizes the power of the large language models to derive theme-aware keywords for items”.
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the large-language model taught by Maragheh with the image cropping method taught by Cheng, Kon, and Saa-Garcia. The motivation for doing so would be to improve the quality of keywords extracted. Maragheh teaches on pg. 4319, Section I., “We discuss how each stage of the framework helps improve the quality of the output keywords and reduce hallucinations.” Thus, it would have been obvious to combine the LLM taught by Maragheh with the image-cropping method taught by Cheng, Kon, and Saa-Garcia in order to obtain the invention as claimed in Claim 19.
Claim(s) 12 is rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al. (US Pub No 20240312020), hereinafter Cheng in view of Kon (US Pub No 20090313141), hereinafter Kon, further in view of Maragheh et al., (R. Y. Maragheh et al., "LLM-TAKE: Theme-Aware Keyword Extraction Using Large Language Models," 2023 IEEE International Conference on Big Data (BigData), Sorrento, Italy, 2023, pp. 4318-4324), hereinafter Maragheh, further in view of Saa-Garcia et al. (US Pub No 20230036950), hereinafter Saa-Garcia, and further in view of Goyal et al. (US Pat No 12033348), hereinafter Goyal.
As to Claim 12, Cheng in view of Kon, Maragheh, and Saa-Garcia fails to teach that the set of digital assets comprises one or more videos.
However, Goyal teaches an image cropping method (see abstract), wherein the digital asset to be cropped is a video frame (Col. 1, lines 22-36, “The memory stores instructions to cause the processor to receive a video stream including a series of video frames…The memory also stores instructions to cause the processor to generate at least one image that depicts the object and that includes a cropped portion of a video frame from the series of video frames”).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the image-cropping method taught by Cheng in view of Kon in order to crop video data as taught by Goyal. The motivation for doing so would be to obtain cropped images of objects from video surveillance data. Goyal teaches in Col. 1, lines 13-17, “Object detection can include the detection of depicted objects such as people and license plates. Applications of object detection include, for example, video surveillance”. Thus, it would have been obvious to combine the teachings of Goyal with the teachings of Cheng and Kon in order to obtain the invention as claimed in Claim 12.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al. (US Pub No 20240312020), hereinafter Cheng in view of Kon (US Pub No 20090313141), hereinafter Kon, and further in view of Maragheh et al., (R. Y. Maragheh et al., "LLM-TAKE: Theme-Aware Keyword Extraction Using Large Language Models," 2023 IEEE International Conference on Big Data (BigData), Sorrento, Italy, 2023, pp. 4318-4324), hereinafter Maragheh, and further in view of Saa-Garcia (US Pub No 20230036950), hereinafter Saa-Garcia, and further in view of Tang et al. (US Pub No 20130101210), hereinafter Tang.
As to Claim 18, Cheng in view of Kon, Maragheh, and Saa-Garcia fails to teach that desired objects are assigned ranking that is a positive number; and restricted objects are assigned a ranking that is a negative number.
Saa-Garcia teaches that objects may be ranked see paragraph [0065], but fails to teach positive or negative rankings. Maragheh teaches that object keywords may be ranked from 0-1 (see Maragheh, pg. 4321, Fig. 4), but fails that to teach that restricted objects are assigned to a negative ranking value.
However, in an analogous art, Tang teaches a method for cropping images (see abstract),
that desired or salient objects are assigned ranking that is a positive number; and restricted objects are assigned a ranking that is a negative number (see paragraph [0023], “For example, if T is 0.3, then the new saliency value range is between −0.3 and 0.7, with −0.3 being a completely irrelevant pixel and 0.7 being an absolutely relevant pixel”).
Thus, it would have been obvious to combine the positive and negative ranking values taught by Tang with the teachings of Cheng, Kon, Maragheh, and Saa-Garcia. The motivation for doing so would be to determine a boundary for the cropped image. Tang teaches in paragraph [0021], “According to certain illustrative examples, to determine a potential cropping boundary, saliency detection function is first applied to an image to create a saliency map. Saliency of a pixel depends on how much that pixel differs from other pixels. Saliency generally corresponds to relevancy of a pixel. The saliency map defines the saliency of each pixel with a saliency value.” Thus, it would have been obvious to combine the positive and negative ranking taught by Tang with the Cheng, Kon, Maragheh, and Saa-Garcia.
Claim(s) 20 is rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al. (US Pub No 20240312020), hereinafter Cheng in view of Kon (US Pub No 20090313141), hereinafter Kon, further in view of Maragheh et al., (R. Y. Maragheh et al., "LLM-TAKE: Theme-Aware Keyword Extraction Using Large Language Models," 2023 IEEE International Conference on Big Data (BigData), Sorrento, Italy, 2023, pp. 4318-4324), hereinafter Maragheh, further in view of Saa-Garcia et al. (US Pub No 20230036950), hereinafter Saa-Garcia, and further in view of Ren et al., (Ren, Tianhe, et al. “Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks.” ArXiv.org, 25 Jan. 2024”), hereinafter Ren.
As to Claim 20, Cheng in view of Kon, Maragheh, and Saa-Garcia fails to teach n the regions of the selected image are identified using a grounded segment anything model that generates a list of identified regions and assigns a ranking to the identified regions of the list of identified regions. Saa-Garcia teaches a model to identify and rank regions (see paragraph[0064],), but fails to teach using a grounded segment anything model.
However, Ren teaches using a Grounded SAM system (see pg. 1, Abstract, “We introduce Grounded SAM, which uses Grounding DINO [38] as an open-set object detector to combine with the segment anything model (SAM)”),
to segment an image and generate a list of identified regions (see pgs. 3-4, Section 3.2., “Given an input image and a text prompt, we first employ Grounding DINO to generate precise boxes for objects or regions within the image by leveraging the textual information as condition. Subsequently, the annotated boxes obtained through Grounding DINO serve as the box prompts for SAM to generate precise mask annotations”).
Thus, it would have been obvious to one of ordinary skill in the art to substitute the identification model taught by Saa-Garcia with the SAM model taught by Ren. The motivation for doing so would be to use this model to handle intricate segmentation tasks. Ren teaches on pg. 2, Section 1, “Although there have been advances in addressing open world tasks through these methodologies, a robust pipeline capable of supporting complex and fundamental open-world tasks such as open-set segmentation is still lacking in the market… Grounded SAM offers a powerful and comprehensive platform that further facilitates an efficient fusion of different expert models to tackle more intricate open-world tasks.” Thus, it would have been obvious to combine the Grounded SAM model taught by Ren with the teachings of Cheng, Kon, Maragheh, and Saa-Garcia in order to obtain the invention as claimed in Claim 20.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Mishra et al. (US Pub No 20250094711) discloses a model which can extract keywords from a content brief, and combine lists of keywords using a model. Mishra teaches that keywords lists can be augmented by removing keywords.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SOUMYA THOMAS whose telephone number is (571)272-8639. The examiner can normally be reached M-F 8:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.T./ Examiner, Art Unit 2664
/JENNIFER MEHMOOD/ Supervisory Patent Examiner, Art Unit 2664
Prosecution Insights