DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 4/11/2024 and 6/3/2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Status of Claims
Claims 1-20 are pending in this application.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Specifically, the Examiner can find no support in the original disclosure for the limitation “wherein the portion pointed to by the cursor is resizable in real- time”. The term resizable does not occur in the Specification or the claims are originally filed.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “real-time” in claims 1, 9 and 17 is a relative term which renders the claim indefinite. The term “real-time” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. Specifically, the term “real-time”, while commonly understood to be very fast, has no commonly accepted upper bound on where the transition from in “real-time” to not in “real-time” occurs. Therefore, the claims are indefinite because the metes and bounds of the claim are unclear.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Ramachandra Iyer et al. (U.S. Patent Application Publication 2019/0370035) in view of Chen et al. (Non-Patent Literature “Shikra: Unleashing Multimodal LLM’s Referential Dialogue Magic”, listed in IDS dated 4/11/2024) in further view of Rangarajan Sridhar et al. (U.S. Patent Application Publication 2022/0093088).
As per claims 1, 9 and 17, Ramachandra Iyer et al. discloses:
A computer system comprising:
a processor set (Figure 1, item 115 and paragraph [0023]);
a set of one or more computer-readable storage media (Figure 1, item 113 and paragraph [0023]);
program instructions, collectively stored in the set of one or more computer-readable storage media (Paragraph [0065] – program components are stored in memory), for causing the processor set to perform the following computer operations:
receive a non-text visual cue and natural language instruction regarding a scene (Paragraphs [0023] & [0040] – the user input may be a query, statement, images, videos, actions, gestures, etc.);
convert the non-text visual cue into a textual location information indicating a portion of an image representing the scene (Paragraph [0040] – objects in images are mapped based on non-textual input with both a location and description); and
trigger using as input at least the textual location information, the natural language instruction, and the image representing the scene, then outputting a response to the input (Claim 1).
Ramachandra Iyer et al. fails to disclose but Chen et al. in the same field of endeavor teaches:
running of a language machine learning model using as input at least the textual location information, the natural language instruction, and the image representing the scene, the language machine learning model outputting a response to the input (Figures 1 & 2 and sections 2.1-2.3 – the LLM is run based on the image, position data and a natural language instruction)
It would be obvious for a person having ordinary skill in the art at the effective filing date of the invention to modify the method, system and computer program product of Ramachandra Iyer et al. with the language machine learning model of Chen et al. because it is a case of simple substitution of one known element (LLM in Chen et al.) for another (generic program in Ramachandra Iyer et al.) to obtain predictable results.
The combination fails to explicitly disclose but Rangarajan Sridhar et al. in the same field of endeavor teaches:
the non-text visual cue includes a cursor pointing to a portion of an image representing the scene, wherein the portion pointed to by the cursor is resizable in real-time (Paragraphs [0058] & [0189] – the cursor selects the variable sized object that is selected (i.e. resizing the portion in real-time))
It would be obvious for a person having ordinary skill in the art at the effective filing date of the invention to modify the method, system and computer program product of Ramachandra Iyer et al. and Chen et al. with the cursor based selection of Rangarajan Sridhar et al. because it is a case of simple substitution of one known element (cursor based selection in Rangarajan Sridhar et al.) for another (drawing of bounding boxes in Ramachandra Iyer et al.) to obtain predictable results.
Claim 1 is directed to the method of using the system of claim 17, so is rejected for similar reasons.
Claim 9 is directed to a computer program product containing instructions to cause a processor to act as the system of claim 17, so is rejected for similar reasons.
As per claims 2, 10 and 18, the combination of Ramachandra Iyer et al., Chen et al. and Rangarajan Sridhar et al. discloses all of the limitations of claims 1, 9 and 17 above. Chen et al. in the combination further discloses:
the non-text visual cue includes a bounding box in the image representing the scene; and wherein the converting of the non-text visual cue into a textual location information includes running a machine learning model that associates the bounding box with semantic information obtained for an area contained in the bounding box and that outputs location coordinates of the bounding box with respect to the image (Figures 1 & 2 and sections 2.1-2.3 – the LLM is run based on the image, position data and a natural language instruction. There are bounding boxes on the images around objects that have coordinates in the image).
As per claims 3 and 11, the combination of Ramachandra Iyer et al., Chen et al. and Rangarajan Sridhar et al. discloses all of the limitations of claims 2 and 10 above. Chen et al. in the combination further discloses:
the machine learning model includes an image segmentation model that segments objects in a given image, correlates bounding boxes of the segmented objects in the given image with respective textual location coordinates of the bounding boxes, and outputs a descriptive text describing the segmented objects via the textual location coordinates (Figures 1 & 2 and sections 2.1-2.3 – the LLM is run based on the image, position data and a natural language instruction. There are bounding boxes on the images around objects that have coordinates in the image and descriptive text associated).
As per claims 4, 12 and 19, the combination of Ramachandra Iyer et al., Chen et al. and Rangarajan Sridhar et al. discloses all of the limitations of claims 1, 9 and 17 above. Chen et al. in the combination further discloses:
the non-text visual cue includes a pointer pointing to an area of the scene; and wherein the converting of the non-text visual cue into the textual location information includes: determining a bounding box that bounds the area of the scene pointed to by the pointer; and running a machine learning model that associates the bounding box with semantic information obtained for an area contained in the bounding box and that outputs location coordinates of the bounding box with respect to the image (Figures 1 & 2 and sections 2.1-2.3 – the LLM is run based on the image, position data and a natural language instruction. There are bounding boxes on the images around objects that have coordinates in the image and descriptive text associated. The objects can be indicated through pointing).
As per claims 5 and 13, the combination of Ramachandra Iyer et al., Chen et al. and Rangarajan Sridhar et al. discloses all of the limitations of claims 4 and 12 above. Chen et al. in the combination further discloses:
the machine learning model includes an image segmentation model that segments objects in a given image, correlates bounding boxes of the segmented objects in the given image with respective textual location coordinates of the bounding boxes, and outputs descriptive text describing the segmented objects via the textual location coordinates (Figures 1 & 2 and sections 2.1-2.3 – the LLM is run based on the image, position data and a natural language instruction. There are bounding boxes on the images around objects that have coordinates in the image and descriptive text associated. The objects can be indicated through pointing).
As per claims 6, 14 and 20, the combination of Ramachandra Iyer et al., Chen et al. and Rangarajan Sridhar et al. discloses all of the limitations of claims 1, 9 and 17 above. Chen et al. in the combination further discloses:
the language machine learning model includes an image encoder and a text encoder, the language machine learning model having been trained based on first embeddings encoded by the image encoder and second embeddings encoded by the text encoder to relate semantic information appearing in sample images to location information incorporated in sample natural language instructions associated with the sample images (Figures 1 & 2 and sections 2.1-2.3 – the multimodal LLM is run based on the image, position data and a natural language instruction with separate embedding for each input.)
As per claims 7 and 15, the combination of Ramachandra Iyer et al., Chen et al. and Rangarajan Sridhar et al. discloses all of the limitations of claims 1 and 9 above. Ramachandra Iyer et al. in the combination further discloses:
the non-text visual cue is obtained via a human-computer interaction performed via a smartphone (Paragraph [0023]).
As per claims 8 and 16, the combination of Ramachandra Iyer et al., Chen et al. and Rangarajan Sridhar et al. discloses all of the limitations of claims 1 and 9 above. Chen et al. in the combination further discloses:
the response is an answer to a question of the input (Figures 1 & 2 and sections 2.1-2.3 – the responses are answers).
Examiner Notes
The Examiner cites particular columns and line numbers in the references as applied to the claims above for the convenience of the Applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested that, in preparing responses, the Applicant fully considers the references in its entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or as disclosed by the Examiner.
Communications via Internet e-mail are at the discretion of the applicant and require written authorization. Should the Applicant wish to communicate via e-mail, including the following paragraph in their response will allow the Examiner to do so:
“Recognizing that Internet communications are not secure, I hereby authorize the USPTO to communicate with me concerning any subject matter of this application by electronic mail. I understand that a copy of these communications will be made of record in the application file.”
Should e-mail communication be desired, the Examiner can be reached at Edwin.Leland@USPTO.gov
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EDWIN S LELAND III whose telephone number is (571)270-5678. The examiner can normally be reached 8:00 - 5:00 M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hai Phan can be reached at 571-272-6338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EDWIN S LELAND III/Primary Examiner, Art Unit 2654