DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-21 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without a practical application or significantly more.
Regarding claims 1, 2, 16 and 19, these claims recite the following limitations which are found to be abstract ideas not reciting a practical application or significantly more, with claim 1 being exemplary:
receiving multi-modal labels corresponding to an image; (abstract idea as a mental process, human mind mentally labeling an image)
wherein the multi-modal labels are generated by two or more pre-trained domain-specific neural network models; (abstract idea as a mental process, human mind mentally determining what the labels of the image are)
mapping the multi-modal labels to a fixed length sequence of multi-modal features; (abstract idea as a mental process, human mind mentally determining what the labels are and limiting how many to associate the image with)
processing the image and the fixed length sequence of multi-modal features by a vision encoder and language decoder to predict text corresponding to the image; and (abstract idea as a mental process, human mind looks at the image and predicts what the text should be corresponding to the image)
This judicial exception is not integrated into a practical application for the following reasons. Claims 1, 16 and 19 further recite additional elements: claim 1 contains “providing the predicted text to a device for presentation to a user”. Claim 16 is directed towards a system, a memory and a processor. Claim 19 is directed towards a non-transitory computer readable storage medium, and a processor.
While the method of Claim 1 further contains, “providing the predicted text to a device for presentation to a user” Although not a complete mental process, this step can be performed by a user with the assistance of pen and paper. While the system of claim 16, a non-transitory computer readable storage medium of claims 19, and the processor of claim 16 and 19 are additional elements, they are not sufficient to recite a practical application of the abstract ideas recited in claims 1,8 and 15 as they amount to mere generic computer elements and thus amount to no more than a recitation of the words "apply it" (or an equivalent) or are no more than mere instructions to implement an abstract idea or other exception on a computer. see MPEP §2106.05(f).
Further, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because when considered separately and in combination, the above recited additional elements from claims 1, 2, 16 and 19 not add significantly more (also known as an “inventive concept”) to the exception. Rather, the claimed “non-transitory computer-readable storage medium” and “processor” perform well-understood, routine, conventional computer functions as recognized by the court decisions listed in MPEP § 2106.05(d).
The dependent claims are also directed to an abstract idea such that the elements can be done mentally.
The claims do not recite additional elements that integrate the judicial exception into a practical application because these additional elements in the claim do no more than automate the mental process that a person may perform.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 7-8, 12-16, 18-19 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. US2024/0161520 hereinafter referred to as Li in view of Lin et al. US2022/0301298 hereinafter referred to as Lin, Bufi US2024/0087303 and Jiang et al. US2019/0377979 hereinafter referred to as Jiang.
As per Claim 1, Li teaches a computer-implemented method, comprising:
processing the image and the features by a vision encoder and language decoder to predict text corresponding to the image; and (Li, Figure 4, 105a, 106, 130b, 415, Paragraph [0051], “FIG. 4A adopts a decoder-based LLM 130b. For decoder-based LLMs 130b, the LLM decoder 130b receives the projected embedding 412, and generate a decoded output text 415 conditioned on the visual representation 412 from the query Transformer 120. Specifically, the decoded output text 415 may be decoded token by toke, and previously generated tokens may be fed to the LLM decoder 130b such that the next token in the output text 415 may be generated conditioned on both the projected embedding 412 and the previously decoded tokens.)
providing the predicted text to a device for presentation to a user. (Li, Paragraph [0051], “FIG. 4A adopts a decoder-based LLM 130b. For decoder-based LLMs 130b, the LLM decoder 130b receives the projected embedding 412, and generate a decoded output text 415 conditioned on the visual representation 412 from the query”)
Li does not explicitly teach receiving multi-modal labels corresponding to an image,
Lin teaches receiving multi-modal labels corresponding to an image, (Lin , Paragraph [0069], “a respective label for each of a plurality of different computer vision tasks (step 302). A “label” for a given image for a given computer vision task is an output that should be generated by a neural network that is configured to perform the given computer vision task by processing the given image. Thus, each training image has multiple labels, one for each of multiple different computer vision tasks” and Paragraph [0024]-[0030], “discloses different tasks that will provide different labels”)
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Li into Lin because by utilizing multi-modal labels instead of standard labels of Lin will provide various different types of labels for the model to use to process for use with different applications.
Li in view of Lin does not explicitly teach wherein the multi-modal labels are generated by two or more pre-trained domain-specific neural network models;
Bufi teaches wherein the multi-modal labels are generated by two or more pre-trained domain-specific neural network models; (Bufi, Paragraph [0061], “The AI visual inspection device 12 may include multiple object detection models. Each object detection model may be a model trained to perform a particular object detection task”)
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Bufi into Li in view of Lin because by separating the different task units from one entity into separate units will result in the same operation of Lin and will allow parallel processing to execute the tasks.
Li in view of Lin and Bufi does not explicitly teach mapping the multi-modal labels to a fixed length sequence of multi-modal features;
Jiang teaches mapping the multi-modal labels to a fixed length sequence of multi-modal features; (Jiang, Paragraph [0055], Figure 4, multi-mode feature vector are determined by the use of feature extraction model and matching model)
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Jiang into Li in view of Lin and Bufi because by using a vector of length will provide a vehicle to store various information in a organized manner for future processing.
Therefore it would have been obvious to one of ordinary skill to combine the four references to obtain the invention in Claim 1.
As per Claim 2, Li teaches a computer-implemented method, comprising:
processing the image and the features by a vision encoder and language decoder to predict text corresponding to the image; and (Li, Figure 4, 105a, 106, 130b, 415, Paragraph [0051], “FIG. 4A adopts a decoder-based LLM 130b. For decoder-based LLMs 130b, the LLM decoder 130b receives the projected embedding 412, and generate a decoded output text 415 conditioned on the visual representation 412 from the query Transformer 120. Specifically, the decoded output text 415 may be decoded token by toke, and previously generated tokens may be fed to the LLM decoder 130b such that the next token in the output text 415 may be generated conditioned on both the projected embedding 412 and the previously decoded tokens.)
Li does not explicitly teach receiving multi-modal labels corresponding to an image,
Lin teaches receiving multi-modal labels corresponding to an image, (Lin , Paragraph [0069], “a respective label for each of a plurality of different computer vision tasks (step 302). A “label” for a given image for a given computer vision task is an output that should be generated by a neural network that is configured to perform the given computer vision task by processing the given image. Thus, each training image has multiple labels, one for each of multiple different computer vision tasks” and Paragraph [0024]-[0030], “discloses different tasks that will provide different labels”)
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Li into Lin because by utilizing multi-modal labels instead of standard labels of Lin will provide various different types of labels for the model to use to process for use with different applications.
Li in view of Lin does not explicitly teach wherein the multi-modal labels are generated by two or more pre-trained domain-specific neural network models;
Bufi teaches wherein the multi-modal labels are generated by two or more pre-trained domain-specific neural network models; (Bufi, Paragraph [0061], “The AI visual inspection device 12 may include multiple object detection models. Each object detection model may be a model trained to perform a particular object detection task”)
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Bufi into Li in view of Lin because by separating the different task units from one entity into separate units will result in the same operation of Lin and will allow parallel processing to execute the tasks.
Li in view of Lin and Bufi does not explicitly teach mapping the multi-modal labels to a fixed length sequence of multi-modal features;
Jiang teaches mapping the multi-modal labels to a fixed length sequence of multi-modal features; (Jiang, Paragraph [0055], Figure 4, multi-mode feature vector are determined by the use of feature extraction model and matching model)
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Jiang into Li in view of Lin and Bufi because by using a vector of length will provide a vehicle to store various information in a organized manner for future processing.
Therefore it would have been obvious to one of ordinary skill to combine the four references to obtain the invention in Claim 2.
As per Claim 3, Li in view of Lin, Bufi and Jiang teaches the computer-implemented method of claim 2, wherein one of the pre-trained domain-specific neural network models is pre-trained to generate at least one of depth data, surface normal vector data, edge data, object labels, segmentation labels, or text labels. (Lin, Paragraph [0024]-[0030], “discloses different tasks that will provide different labels” and Bufi, Paragraph [0061])
The rationale applied to the rejection of claim 2 has been incorporated herein.
As per Claim 7, Li in view of Lin, Bufi and Jiang teaches the computer-implemented method of claim 2, wherein the predicted text is a caption corresponding to the image. (Li, Paragraph [0030], Figure 1)
The rationale applied to the rejection of claim 2 has been incorporated herein.
As per Claim 8, Li in view of Lin, Bufi and Jiang teaches the computer-implemented method of claim 2, wherein a question about content depicted in the image is input to the language decoder and the predicted text is an answer to the question. (Li, Paragraph [0025] and [0030], Figure 1)
The rationale applied to the rejection of claim 2 has been incorporated herein.
As per Claim 12, Li in view of Lin, Bufi and Jiang teaches the computer-implemented method of claim 2, wherein at least one of the steps of receiving, mapping, and processing is performed on a server or in a data center to generate the text corresponding to the image and is streamed to a user device. (Li, Paragraph [0060], “Consistent with such embodiments, processor 610 and/or memory 620 may be located in one or more data centers and/or cloud computing facilities”, Paragraph [0067]-[0068])
The rationale applied to the rejection of claim 2 has been incorporated herein.
As per Claim 13, Li in view of Lin, Bufi and Jiang teaches the computer-implemented method of claim 2, wherein at least one of the steps of receiving, mapping, and processing is performed within a cloud computing environment. (Li, Paragraph [0060], “Consistent with such embodiments, processor 610 and/or memory 620 may be located in one or more data centers and/or cloud computing facilities”)
The rationale applied to the rejection of claim 2 has been incorporated herein.
As per Claim 14, Li in view of Lin, Bufi and Jiang teaches the computer-implemented method of claim 2, wherein at least one of the steps of receiving, mapping, and processing is performed for training, testing, or certifying a neural network employed in a machine, robot, or autonomous vehicle. (Li, Paragraph [0070])
The rationale applied to the rejection of claim 2 has been incorporated herein.
As per Claim 15, Li in view of Lin, Bufi and Jiang teaches the computer-implemented method of claim 2, wherein at least one of the steps of receiving, mapping, and processing is performed on a virtual machine comprising a portion of a graphics processing unit. (Li, Paragraph[0058])
The rationale applied to the rejection of claim 2 has been incorporated herein.
As per Claim 16, Claim 16 claims a system comprising a processor (Lin, Paragraph [0090]) configured to perform the method as claimed in Claim 2. Therefore the rejection and rationale are analogous to that made in Claim 2.
As per Claim 18, Claim 18 claims the same limitation as Claim 3 and is dependent on a similarly rejected independent claim. Therefore the rejection and rationale are analogous to that made in Claim 3.
As per Claim 19, Claim 19 claims a non-transitory computer-readable media storing computer instructions that, when executed by one or more processors, cause the one or more processors (Lin, Paragraph [0090]) perform the method as claimed in Claim 2. Therefore the rejection and rationale are analogous to that made in Claim 2.
As per Claim 21, Claim 21 claims the same limitation as Claim 3 and is dependent on a similarly rejected independent claim. Therefore the rejection and rationale are analogous to that made in Claim 3.
Claims 5, 17 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. US2024/0161520 hereinafter referred to as Li in view of Lin et al. US2022/0301298 hereinafter referred to as Lin, Bufi US2024/0087303 and Jiang et al. US2019/0377979 hereinafter referred to as Jiang as applied to Claims 2, 16 and 19 respectively and further in view of Piergiovanni et al. US2024/0119713 hereinafter referred to as Piergiovanni.
As per Claim 5, Li in view of Lin, Bufi and Jiang teaches the computer-implemented method of claim 2, wherein the vision encoder
Li in view of Lin, Bufi and Jiang does not explicitly teach comprises an adaptor configured to smooth transitions from domain-specific vision features to language features.
Piergiovanni teaches comprises an adaptor configured to smooth transitions from domain-specific vision features to language features. (Piergiovanni, Paragraph [0031] and Figure 1, “Besides concatenation, other methods use multimodal transformers composed of both self-attention and cross-attention in every block.for the image and text tokens respectively. Next, the computing system can employ some number (e.g., only two) of cross-attention layers (e.g., unlike co-attention that uses cross-attention and self-attention in every block) to create visual and language compound tokens”)
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Piergiovanni into Li in view of Lin, Bufi and Jiang because by utilizing cross-attention layers will provide an improved vision language model.
Therefore it would have been obvious to one of ordinary skill to combine the five references to obtain the invention in Claim 5.
As per Claim 17, Claim 17 claims the same limitation as Claim 5 and is dependent on a similarly rejected independent claim. Therefore the rejection and rationale are analogous to that made in Claim 5.
As per Claim 20, Claim 20 claims the same limitation as Claim 5 and is dependent on a similarly rejected independent claim. Therefore the rejection and rationale are analogous to that made in Claim 5.
Claims 10-11 are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. US2024/0161520 hereinafter referred to as Li in view of Lin et al. US2022/0301298 hereinafter referred to as Lin, Bufi US2024/0087303 and Jiang et al. US2019/0377979 hereinafter referred to as Jiang as applied to Claim 2 and further in view of Kumar et al. US2023/0199402 hereinafter referred to as Kumar.
As per Claim 10, Li in view of Lin, Bufi and Jiang teaches the computer-implemented method of claim 2,
Li in view of Lin, Bufi and Jiang does not explicitly teach further comprising adjusting parameters applied by the mapping to reduce differences between the predicted text and reference text associated with the image.
Kumar teaches further comprising adjusting parameters applied by the mapping to reduce differences between the predicted text and reference text associated with the image. (Kumar, Paragraph [0087], “For example, a cross-domain classification machine learning model may describe a machine learning model that has been pre-trained based at least in part on embeddings associated with source domain training data and target domain training data, and has been fine-tuned using source domain training data (e.g., labeled source domain training data) based at least in part on a contrastive loss function as further described herein. The fine-tuned machine learning model that is the cross-domain classification machine learning model may be configured to generate text classification predictions for reference text data objects originating from or otherwise associated with a target domain (e.g., target domain reference text data objects) that is different from the source domain, wherein generating text classification predictions may comprise assigning target text data objects to the noted reference text data objects”)
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Kumar into Li in view of Lin, Bufi and Jiang because by utilizing loss function to adjust parameters will result in improving the accuracy of the model.
Therefore it would have been obvious to one of ordinary skill to combine the five references to obtain the invention in Claim 10.
As per Claim 11, Li in view of Lin, Bufi, Jiang and Kumar teaches the computer-implemented method of claim 10, wherein the differences are reduced according to a loss function. (Kumar, Paragraph [0087], “For example, a cross-domain classification machine learning model may describe a machine learning model that has been pre-trained based at least in part on embeddings associated with source domain training data and target domain training data, and has been fine-tuned using source domain training data (e.g., labeled source domain training data) based at least in part on a contrastive loss function as further described herein.”)
The rationale applied to the rejection of claim 10 has been incorporated herein.
Allowable Subject Matter
Claims 4, 6 and 9 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MING HON whose telephone number is (571)270-5245. The examiner can normally be reached M-F 9am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached on 571-270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MING Y HON/Primary Examiner, Art Unit 2666