DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 2, 5-7, 10, 13-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xie et al. (U.S. Patent Application 20230394855) in view of Turock et al. (U.S. Patent Application 20240212804).
In regards to claim 1, Xie teaches a method [Fig. 4A; e.g. method, 0050] implemented by one or more processors [e.g. one or more processors, 0084], the method comprising:
receiving a request, wherein the request includes image data corresponding to an image that includes one or more objects [e.g. Receive, by a first vision language model, a request for a story caption of an image, the request comprising a caption focus for the image. Object detector 116 detects a plurality of objects 118 in image 102, see claim 1 of Xie, 0019];
providing the updated image data to one or more image analysis modules [Fig. 1; e.g. The plurality of detected objects is provided to the vision language model. Updating the image data was performed in Turock as explained below. Furthermore, using Turock’s updating, one or more detected objects are redacted, 0019-0020], wherein one or more of the image analysis modules are configured to process the updated image data to generate textual output that is representative of one or more of the objects included in the updated image data [Fig. 1; e.g. Visual information 120 comprises text that describes what is contained within image 102, such as image tags 122, an initial image caption 124, and object information 126. Using Turock’s updated image data (see below), Xie generates text on only the non-redacted objects contained within the image, 0020];
receiving, in response to providing the updated image data to one or more of the image analysis modules and from one or more of the image analysis modules [e.g. after providing the plurality of detected objects (using Turock’s updating, see below) to the vision language model, 0019-0020], the textual output [Fig. 1; e.g. the generative language model receives the visual information comprising text, 0021];
generating, based on the textual output received from one or more of the image analysis modules, an input prompt for a generative model [e.g. A rich semantic representation of an input image, such as image tags, object attributes and locations, and captions, is constructed as a structured textual prompt, termed “visual clues”, using a vision foundation model for the generative language model, 0018, 0021];
providing the input prompt to the generative model, wherein providing the input prompt to the generative model causes processing of the input prompt using the generative model [e.g. structured visual clues 130, which is used to directly prompt generative language model 140, 0040];
receiving, in response to providing the input prompt, generative content that is generated based on processing the input prompt using the generative model [e.g. Based on the visual clues, a large language model, which is able to generate long, coherent paragraphs, is used to produce a series of comprehensive descriptions for the visual content. A closed-loop verification procedure, passing plurality of image story caption candidates 144, output from generative language model 140, through vision language model 150, 0018, 0026]; and
causing the generative content to be provided responsive to the request [e.g. The annotated image is provided to a text-to-speech interface. Receive, by a first vision language model, a request for a story caption of an image, the request comprising a caption focus for the image, 0048, also see claim 1 of Xie].
Xie does not explicitly teach
one or more objects in an environment (emphasis added);
determining, based on the image data, that a particular object of the one or more objects has one or more particular features and/or has a particular classification;
in response to determining the particular object has one or more of the particular features and/or has the particular classification, generating updated image data, wherein generating the updated image data comprises generating the updated image data to exclude the particular object.
However, Turock teaches
one or more objects in an environment [e.g. range of objects in the environment, 0058];
determining, based on the image data, that a particular object of the one or more objects has one or more particular features and/or has a particular classification [e.g. Detected Objects 401 represent the class or type of the object that was detected, 0070];
in response to determining the particular object has one or more of the particular features and/or has the particular classification [e.g. after detecting the objects class or type, 0070], generating updated image data, wherein generating the updated image data comprises generating the updated image data to exclude the particular object [e.g. The redaction can be applied by replacing person pixels in an image data with black pixels or blurring or distorting the pixels, 0059].
Therefore, it would have been obvious to one of ordinary skill in the art to have modified Xie’s method with the features of
one or more objects in an environment;
determining, based on the image data, that a particular object of the one or more objects has one or more particular features and/or has a particular classification;
in response to determining the particular object has one or more of the particular features and/or has the particular classification, generating updated image data, wherein generating the updated image data comprises generating the updated image data to exclude the particular object
in the same conventional manner as taught by Turock because Turock provides a method where the privacy of an individual is maintained at any situation [0059].
In regards to claim 2, Xie teaches the method of claim 1, further comprising:
prior to generating the input prompt, filtering the textual output from one or more of the image analysis modules [e.g. filter the output at the sentence level by splitting it into sentences (s.sub.1, s.sub.2, . . . , s.sub.U), and using a threshold γ to remove the sentences with lower similarities to obtain the final output T*, 0043] to remove one or more descriptions [e.g. to remove the sentences, 0043] of the one or more objects having one or more of the particular features and/or having the particular classification [e.g. if a caption includes a sentence “The man wears a green shirt” and the image contains a man wearing a blue shirt and a green bush, if the green bush is not in the same cropped portion of the image as the man in the blue shirt, the sentence will be filtered out, 0049].
In regards to claim 5, Xie does not explicitly teach the method of claim 1, wherein generating the updated image data that excludes the particular object further comprises: obfuscating the one or more objects having one or more of the particular features and/or having the particular classification.
However, Turock teaches the method of claim 1, wherein generating the updated image data that excludes the particular object [see rejection of claim 1 above] further comprises: obfuscating the one or more objects having one or more of the particular features and/or having the particular classification [e.g. distorting the region, blurring by adding mosaic censorship, 0087].
Therefore, it would have been obvious to one of ordinary skill in the art to have modified Xie’s method with the features of
obfuscating the one or more objects having one or more of the particular features and/or having the particular classification
in the same conventional manner as taught by Turock because Turock provides a method where the privacy of an individual is maintained at any situation [0059].
In regards to claim 6, Xie does not explicitly teach the method of claim 1, wherein obfuscating the one or more objects having one or more of the particular features and/or having the particular classification comprises: altering one or more pixel values, wherein the one or more pixel values correspond to color settings of one or more pixels of the image data.
However, Turock teaches the method of claim 1, wherein obfuscating the one or more objects having one or more of the particular features and/or having the particular classification comprises: altering one or more pixel values, wherein the one or more pixel values correspond to color settings of one or more pixels of the image data [e.g. The redaction can be done by replacing person pixels with black pixels, 0087].
Therefore, it would have been obvious to one of ordinary skill in the art to have modified Xie’s method with the features of
wherein obfuscating the one or more objects having one or more of the particular features and/or having the particular classification comprises: altering one or more pixel values, wherein the one or more pixel values correspond to color settings of one or more pixels of the image data
in the same conventional manner as taught by Turock because Turock provides a method where the privacy of an individual is maintained at any situation [0059].
In regards to claim 7, Xie does not explicitly teach the method of claim 1, wherein generating the updated image data that excludes the particular object further comprises:
partitioning the image data into one or more image segments, wherein a particular image segment of the one or more image segments includes the particular object having one or more of the particular features and/or having the particular classification; and
excluding, from the updated image data, the particular image segment while including one or more other of the image segments.
However, Turock teaches the method of claim 1, wherein generating the updated image data that excludes the particular object further [see rejection of claim 1 above] comprises:
partitioning the image data into one or more image segments, wherein a particular image segment of the one or more image segments includes the particular object having one or more of the particular features and/or having the particular classification [e.g. applies the segmentation model to generate the segmentation mask of the image and from the mask it retrieves the segmented regions of each object, 0098]; and
excluding, from the updated image data, the particular image segment while including one or more other of the image segments [e.g. only the Individual's 1901 Segmented Area 2101 is being redacted, but the Bed 534 and everything else described with respect to FIG. 21A is visible, 0099].
Therefore, it would have been obvious to one of ordinary skill in the art to have modified Xie’s method with the features of
partitioning the image data into one or more image segments, wherein a particular image segment of the one or more image segments includes the particular object having one or more of the particular features and/or having the particular classification; and
excluding, from the updated image data, the particular image segment while including one or more other of the image segments
in the same conventional manner as taught by Turock because Turock provides a method where the privacy of an individual is maintained at any situation [0059].
In regards to claim 10, Xie teaches a method [Fig. 4A; e.g. method, 0050] implemented by one or more processors [e.g. one or more processors, 0084], the method comprising:
receiving a request, wherein the request includes image data corresponding to an image that includes one or more objects [e.g. Receive, by a first vision language model, a request for a story caption of an image, the request comprising a caption focus for the image. Object detector 116 detects a plurality of objects 118 in image 102, see claim 1 of Xie, 0019];
providing the image data to one or more image analysis modules [Fig. 1; e.g. The plurality of detected objects is provided to the vision language model, 0019-0020], wherein one or more of the image analysis modules are configured to process the image data to generate textual output that is representative of one or more of the objects included in the image data [Fig. 1; e.g. Visual information 120 comprises text that describes what is contained within image 102, such as image tags 122, an initial image caption 124, and object information 126, 0020];
receiving, in response to providing the image data to one or more of the image analysis modules and from one or more of the image analysis modules [e.g. after providing the plurality of detected objects to the vision language model, 0019-0020], the textual output [Fig. 1; e.g. the generative language model receives the visual information comprising text, 0021];
generating filtered textual output [e.g. filter the output at the sentence level by splitting it into sentences (s.sub.1, s.sub.2, . . . , s.sub.U), and using a threshold γ to remove the sentences with lower similarities to obtain the final output T*, 0043], wherein generating the filtered textual output comprises generating the filtered the textual output to exclude a portion of the textual output [e.g. to remove the sentences, 0043] that is representative of one or more of the objects having one or more particular features [e.g. if a caption includes a sentence “The man wears a green shirt” and the image contains a man wearing a blue shirt and a green bush, if the green bush is not in the same cropped portion of the image as the man in the blue shirt, the sentence will be filtered out, 0049];
generating, subsequent to generating the filtered textual output, an input prompt for a generative model based on the filtered textual output [e.g. A rich semantic representation of an input image, such as image tags, object attributes and locations, and filtered captions, is constructed as a structured textual prompt, termed “visual clues”, using a vision foundation model for the generative language model, 0018, 0021];
providing the input prompt to the generative model, wherein providing the input prompt to the generative model causes processing of the input prompt using the generative model [e.g. structured visual clues 130, which is used to directly prompt generative language model 140, 0040];
receiving, in response to providing the input prompt, generative content that is generated based on processing the input prompt using the generative model [e.g. Based on the visual clues, a large language model, which is able to generate long, coherent paragraphs, is used to produce a series of comprehensive descriptions for the visual content. A closed-loop verification procedure, passing plurality of image story caption candidates 144, output from generative language model 140, through vision language model 150, 0018, 0026]; and
causing the generative content to be provided responsive to the request [e.g. The annotated image is provided to a text-to-speech interface. Receive, by a first vision language model, a request for a story caption of an image, the request comprising a caption focus for the image, 0048, also see claim 1 of Xie].
Xie does not explicitly teach
one or more objects in an environment (emphasis added).
However, Turock teaches
one or more objects in an environment [e.g. range of objects in the environment, 0058].
Therefore, it would have been obvious to one of ordinary skill in the art to have modified Xie’s method with the features of
one or more objects in an environment;
in the same conventional manner as taught by Turock because Turock provides a method where the privacy of an individual is maintained at any situation [0059].
In regards to claim 13, Xie teaches the method of claim 12, wherein the content being currently rendered at the computing device is visual content [Fig. 2; e.g. viewing the input image 104 comprising a picture as illustrated in Figure 2, 0022].
In regards to claim 14, Xie teaches a method [Fig. 4A; e.g. method, 0050] implemented by one or more processors [e.g. one or more processors, 0084], the method comprising:
receiving a request, wherein the request includes image data corresponding to an image that includes one or more objects [e.g. Receive, by a first vision language model, a request for a story caption of an image, the request comprising a caption focus for the image. Object detector 116 detects a plurality of objects 118 in image 102, see claim 1 of Xie, 0019];
generating an input prompt for a generative model [e.g. A rich semantic representation of an input image, such as image tags, object attributes and locations, and captions, is constructed as a structured textual prompt, termed “visual clues”, using a vision foundation model for the generative language model, 0018, 0021], wherein the input prompt includes the updated image data [e.g. A structured textual prompt is generated using Turock’s redacted image data (see Turock below), 0018, 0021, also see 0059 of Turock];
providing the input prompt, including the updated image data [e.g. A structured textual prompt is generated using Turock’s redacted image data (see Turock below), 0018, 0021, also see 0059 of Turock], to the generative model, wherein providing the input prompt, including the updated image data, to the generative model causes processing of the input prompt, including the updated image data, using the generative model [e.g. structured visual clues 130, which is used to directly prompt generative language model 140, 0040];
receiving, in response to providing the input prompt, including the updated image data, to the generative mode, generative content that is generated based on processing the input prompt, including the updated image data, using the generative model [e.g. Based on the visual clues using Turock’s redacted image data (see Turock below), a large language model, which is able to generate long, coherent paragraphs, is used to produce a series of comprehensive descriptions for the visual content. A closed-loop verification procedure, passing plurality of image story caption candidates 144, output from generative language model 140, through vision language model 150, 0018, 0026, also see 0059 of Turock]; and
causing the generative content to be provided responsive to the request [e.g. The annotated image is provided to a text-to-speech interface. Receive, by a first vision language model, a request for a story caption of an image, the request comprising a caption focus for the image, 0048, also see claim 1 of Xie].
Xie does not explicitly teach
one or more objects in an environment (emphasis added);
determining, based on the image data, that a particular object of the one or more objects has one or more particular features and/or has a particular classification;
in response to determining the particular object has one or more of the particular features and/or has the particular classification, generating updated image data, wherein generating the updated image data comprises generating the updated image data to exclude the particular object.
However, Turock teaches
one or more objects in an environment [e.g. range of objects in the environment, 0058];
determining, based on the image data, that a particular object of the one or more objects has one or more particular features and/or has a particular classification [e.g. Detected Objects 401 represent the class or type of the object that was detected, 0070];
in response to determining the particular object has one or more of the particular features and/or has the particular classification [e.g. after detecting the objects class or type, 0070], generating updated image data, wherein generating the updated image data comprises generating the updated image data to exclude the particular object [e.g. The redaction can be applied by replacing person pixels in an image data with black pixels or blurring or distorting the pixels, 0059].
Therefore, it would have been obvious to one of ordinary skill in the art to have modified Xie’s method with the features of
one or more objects in an environment;
determining, based on the image data, that a particular object of the one or more objects has one or more particular features and/or has a particular classification;
in response to determining the particular object has one or more of the particular features and/or has the particular classification, generating updated image data, wherein generating the updated image data comprises generating the updated image data to exclude the particular object
in the same conventional manner as taught by Turock because Turock provides a method where the privacy of an individual is maintained at any situation [0059].
In regards to claim 15, the claim recites similar limitations as claim 5. Therefore, the same rationale as claim 5 is applied.
In regards to claim 16, the claim recites similar limitations as claim 6. Therefore, the same rationale as claim 6 is applied.
In regards to claim 17, the claim recites similar limitations as claim 7. Therefore, the same rationale as claim 7 is applied.
In regards to claim 18, the claim recites similar limitations as claim 7. Therefore, the same rationale as claim 7 is applied.
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xie et al. (U.S. Patent Application 20230394855) in view of Turock et al. (U.S. Patent Application 20240212804) as applied to claim 2 above, and further in view of Jones et al. (U.S. Patent Application 20250150474).
In regards to claim 3, Xie as modified by Turock does not explicitly teach the method of claim 2, wherein filtering the textual output from one or more of the image analysis modules causes the input prompt to be generated without the one or more descriptions of the one or more objects having one or more of the particular features and/or having the particular classification.
However, Jones teaches the method of claim 2, wherein filtering the textual output from one or more of the image analysis modules causes the input prompt to be generated without the one or more descriptions of the one or more objects having one or more of the particular features and/or having the particular classification [e.g. redacting a portion of data in prompt data that has been identified to be redacted and replaced the portion of data with a redacted data tag, 0045].
Therefore, it would have been obvious to one of ordinary skill in the art to have modified the combination of Xie’s method and the teachings of Turock with the features of wherein filtering the textual output from one or more of the image analysis modules causes the input prompt to be generated without the one or more descriptions of the one or more objects having one or more of the particular features and/or having the particular classification in the same conventional manner as taught by Jones because Jones provides a method for protecting against potential data privacy and data leakage issues for generative AI applications [0002].
Claim(s) 12, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xie et al. (U.S. Patent Application 20230394855) in view of Turock et al. (U.S. Patent Application 20240212804) as applied to claims 10, 14 above, and further in view of Deutsch et al. (U.S. Patent Application 20250104243).
In regards to claim 12, Xie as modified by Turock does not explicitly teach the method of claim 10, wherein the image data corresponding to the image is based on content being currently rendered at the computing device.
However, Deutsch teaches the method of claim 10, wherein the image data corresponding to the image is based on content being currently rendered at the computing device [e.g. the image may be a current screen shot of the GUI state shown to the user, 0049].
Therefore, it would have been obvious to one of ordinary skill in the art to have modified the combination of Xie’s method and the teachings of Turock with the features of wherein the image data corresponding to the image is based on content being currently rendered at the computing device in the same conventional manner as taught by Deutsch because using the content being currently rendered at the computing device is well known and commonly used in the art of computer graphical systems.
In regards to claim 20, the claim recites similar limitations as claim 12. Therefore, the same rationale as claim 12 is applied.
Allowable Subject Matter
Claims 4, 8, 9, 11, 19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
In regards to claim 4, the prior art of record fails to teach or suggest the method of claim 2, wherein filtering the textual output from one or more of the image analysis modules to remove one or more of the descriptions of the one or more objects having one or more of the particular features and/or having the particular classification comprises:
providing, as input to an additional generative model: the textual output from one or more of the image analysis modules; and an additional input prompt, wherein the additional input prompt, wherein the additional input prompt includes instructions that cause one or more of the descriptions of the one or more objects having one more of the particular features and/or having the particular classification to be removed.
In regards to claim 8, the prior art of record fails to teach or suggest the method of claim 7, wherein providing the updated image data to one or more of the image analysis modules further comprises: providing a subset of the one or more image segments to one or more of the image analysis modules, wherein the subset of the one or more image segments excludes the particular image segment that includes the particular object having one or more of the particular features and/or having the particular classification.
In regards to claim 9, the prior art of record fails to teach or suggest the method of claim 1, wherein the input prompt causes omission, in the generative content that is responsive to the user request, of any reference to the particular object having one or more of the particular features and/or having the particular classification.
In regards to claim 11, the claim recites similar limitations as claim 9. Therefore, claim 11 is allowed for at least the same reason as claim 9 if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
In regards to claim 19, the claim recites similar limitations as claim 9. Therefore, claim 19 is allowed for at least the same reason as claim 9 if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW SHIN whose telephone number is (571)270-5764. The examiner can normally be reached Monday - Friday from 11:00AM to 7:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said Broome can be reached at 571-272-2931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW SHIN/Examiner, Art Unit 2612
/Said Broome/Supervisory Patent Examiner, Art Unit 2612
Prosecution Insights