DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 13, 16, 17, 19, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Xiong et al (US 12,333,730).
With respect to claim 1, Xiong discloses a device comprising: a memory configured to store model data associated with a trained multimodal model; and one or more processors coupled to the memory, wherein the one or more processors are configured to:
obtain image data representing an image, (see figure 2, numerical 205);
obtain data representing a region of interest (ROI) within the image, (see figure 2, numerical 210);
determine boundaries of the ROI within the image based on the data, (see col. 13, lines 54-57,wherein …Additionally, an object boundary 815 between “Person 6” and “Person 7” has been determined…);
generate model input data based on the image data and the data, (see figure 2, numerical 205 “image data” and numerical 210 “the data” is combined to generate a input data into fully convolutional network “model”);
[selectively modify the model input data based on the boundaries]; and
provide the model input data as input to the trained multimodal model to generate a response output, (see figure 2, numerical 280 and the input provided to the network).
However, Xiong fails to explicitly disclose selectively modify the model input data based on the boundaries, as claimed.
But, in col. 11 lines 61-65, wherein …an operation 615, for a given ROI …resized to a common size…, this very well be modifying the model input data based on the boundaries of ROI.
Therefore, it would have been obvious to one ordinary skilled in the art at the effective date of invention to utilize the teaching of resizing the ROI to a common size i.e. selectively modify the model input data …boundaries, into the Xiong system to yield the predictable results i.e. an input data for a model to attain a response output.
With respect to claim 2, Xiong further discloses divide the image into a set of tiles, wherein the model input data represents the set of tiles, and wherein each tile of the set of tiles has a corresponding size that is based on a size criterion associated with an image encoding and mapping model, (see col. 8, lines 26-30, wherein …illustrative example of FIG. 3, at an operation 305, image data from the initially-obtained frame(s) of image data (such as the image data 205) is input into the neural network 300 as image patches of size m×m×x…), as claimed.
With respect to claim 3, Xiong further discloses wherein the one or more processors are configured to: determine, based on the boundaries, whether the ROI extends across multiple tiles of the set of tiles, wherein the model input data is modified based on the ROI extending across the multiple tiles, (see figure 8A, person 7 is overlapping between two zones, figure 6, numerical 615 and 620, to Resize the ROIs of feature Maps an ROIs depth maps in order to get a Fixed size of ROIs), as claimed.
With respect to claim 13, Xiong further discloses one or more cameras coupled to the one or more processors and configured to generate the image data, (see figure 1, numerical 186), as claimed.
With respect to claim 16, Xiong further discloses wherein the one or more processors are included in an integrated circuit, (see figure 1, 140 and 160), as claimed.
With respect to claim 17, Xiong further discloses wherein the one or more processors are integrated in at least one of a mobile phone, a tablet computer device, a wearable electronic device, an extended reality (XR) device, or a camera device, and wherein the mobile phone, the tablet computer device, the wearable electronic device, the XR device, or the camera device is configured to output the response output, (see col. 12, lines 39-40, wherein …the components described with reference to FIG. 1 are components of an AR or XR headset…), as claimed.
Claims 19 and 20 are rejected for the same reasons as set forth in the rejections for claim 1, because claims 19 and 20 are claiming subject matter of similar scope as claimed in claim 1.
Claims 5, 6, 10, 11, 12, 14, 15 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Xiong et al (US 12,333,730) in view of Lakhotia et al (US Pub. 2026/0011149).
With respect to claim 10, Xiong further discloses all the elements as claimed and as rejected in claim 1 above. However, Xiong fails to explicitly disclose obtain one or more hyperparameter values of the trained multimodal model, wherein the one or more hyperparameter values are indicative of a relative weighting of features associated with the ROI relative to features of the image for areas outside the ROI, as claimed.
Lakhotia teaches obtain one or more hyperparameter values of the trained multimodal model, wherein the one or more hyperparameter values are indicative of a relative weighting of features associated with the ROI relative to features of the image for areas outside the ROI, (see figure 6, and paragraph 0158, wherein …As such, each encoder can accept a sequence of vectors, passing each vector through the self-attention layer, then the feedforward network, and then upwards to the next encoder in the stack…), as claimed.
It would have been obvious to one ordinary skilled in the art at the effective date of invention to combine the two references as they are analogous because they are solving similar problem of object localization using image analysis. The teaching of Lakhotia to use an attention layer i.e. hyperparameter values, can be incorporated into Xiong as suggested (see Xiong main processor in figure 1), for suggestion, and modifying the system will yield an output as needed by the trained network, (see Lakhotia paragraph 0003), for suggestion.
With respect to claim 5, for the same reasons to combine Xiong and Lakhotia further discloses wherein, prior to modification of the model input data, the model input data represents the image and a query associated with the image, (see Lakhotia figure 1, numerical 104 queries are inputted into the processor with the videos)
With respect to claim 6, for the same reasons to combine Xiong and Lakhotia further discloses wherein the one or more processors are configured to: determine whether the boundaries satisfy one or more thresholds, wherein, after modification of the model input data, the model input data further represents the ROI based on the boundaries satisfying the one or more thresholds, (see Lakhotia paragraph 0079, wherein …For example, a query such as “person entering a building” can be matched to stored captions, and the corresponding image embeddings can be retrieved from the model database 118 and passed to downstream models for further inference and/or presentation…), as claimed.
With respect to claim 11, for the same reasons to combine Xiong and Lakhotia further discloses the trained multimodal model includes an image encoding and mapping model, a text encoding model, and a language model; the image encoding and mapping model is configured to generate first feature data based on the model input data; the text encoding model is configured to generate second feature data based on the model input data; and the language model is configured to generate the response output based on the first feature data and the second feature data, (see Lakhotia paragraph 0154, wherein …In some implementations in which the input 601 includes multimodal data, the embedding component 620 can fuse representations of the different types of data (e.g., text, image, audio, USD, video, design, etc.) using techniques like early fusion (concatenation), late fusion (sequential processing), attention-based fusion (e.g., self-attention, cross-attention), etc.), as claimed.
With respect to claim 12, for the same reasons to combine Xiong and Lakhotia further discloses a modem coupled to the one or more processors and configured to receive the image data, the data representing the ROI, or a combination thereof, (see Lakhotia paragraph 0192, wherein …computing device(s) 700 described herein with respect to FIG. 7. By way of example and not limitation, a client device can be embodied as …a smartphone, a tablet computer … [a smart phone has modem to receive and transmit data]), as claimed.
With respect to claim 14, for the same reasons to combine Xiong and Lakhotia further discloses one or more microphones configured to generate audio data representing user speech, wherein the data representing the ROI includes the audio data, (see Lakhotia paragraph 0192, wherein …computing device(s) 700 described herein with respect to FIG. 7. By way of example and not limitation, a client device can be embodied as …a smartphone, a tablet computer … [a smart phone has microphones for the user to input a speech prompt]), as claimed.
With respect to claim 15, for the same reasons to combine Xiong and Lakhotia further discloses a user interface configured to generate text data based on user input, wherein the data representing the ROI includes the text data, (see Lakhotia paragraph 0192, wherein …computing device(s) 700 described herein with respect to FIG. 7. By way of example and not limitation, a client device can be embodied as …a smartphone, a tablet computer … [a smart phone has a keypad for text input]), as claimed.
With respect to claim 18, for the same reasons to combine Xiong and Lakhotia further discloses wherein the one or more processors are integrated in a vehicle that is configured to output the response output, (see Lakhotia paragraph 0192, wherein …computing device(s) 700 described herein with respect to FIG. 7. By way of example and not limitation, a client device can be embodied as …a vehicle, a boat, a flying vessel, a virtual machine…), as claimed.
Allowable Subject Matter
Claims 4 and 7-9 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VIKKRAM BALI whose telephone number is (571)272-7415. The examiner can normally be reached Monday-Friday 7:00AM-3:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at 571-272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/VIKKRAM BALI/Primary Examiner, Art Unit 2663
Prosecution Insights