DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention, the rejections are as follows:
Claims 1, 5, 15, 17, 18 and 20 variously recite, the word “only”, repeatedly, in the phrase “to only derive room type from a room depicted in an image . . .or a combination thereof . .” is in contradiction with the phrase “or a combination thereof”. Therefore, these citations are vague and confusing because it is unclear what feature or element is further limited by this language. For the purposes of this office the word “only” is not given a patentable weight.
Claims 2-14, 16-17 and 19-20 depend from an indefinite base claim and are thus indefinite themselves.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-9 and 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over US 2023/0359830 A1 to Koh in view of US 11087167 B2 to Das et al (hereinafter ‘Das’).
Regarding claim 1, Koh discloses a system, comprising: a memory storing instructions; and one or more hardware processors configured to execute the instructions (Para [0009], wherein systems comprising at least one computing device comprising at least one processor, a memory, and instructions executable by the at least one processor) to perform operations comprising: selecting a plurality of Artificial Intelligence (AI) models for training in image classification (Para [0040], wherein a base classifier is a classifier (i.e., a machine learning model that is used to make predictions about a particular type of data) trained on a specific set of training data. The complexity of the model structure can vary, with some base classifiers having relatively simple models and others having more complex models such as deep neural networks. The choice of model structure, as selection, will depend on the nature of the data and the specific goals of the task at hand. Because a base classifier is trained on a specific set of training data, it may be specialized in the patterns that are observed in that data. This can make it more effective at making predictions about similar types of data. To take advantage of multiple base classifiers, as plurality of models, an ensemble of base classifiers can be used); training a first AI model of the plurality of AI models to derive an object depicted in an image, based on a first training data set (Para [0043] and [0101], wherein FIG. 3 shows a non-limiting example of a process 300 for training a base classifier by NLP generated semantic maps. The process 300 may begin with operation 302, wherein the system may collect documentations, manuals, or handbooks. For example, the system may identify the maker or manufacturers, as the object, for one or more IoT devices, and wherein data input type may be a video or still image capture device (e.g., a camera)); training a second AI model of the plurality of AI models to only derive the room type from the room depicted in the image, to derive an object (Para [0064], wherein stacking is a type of ensemble method in which a second-level model is trained, as the second AI model, to combine the predictions of multiple base models. The base models can be trained using any method, and the second-level model can be trained using supervised learning techniques, e.g. Mixture of Experts (ME)), based on a second data set having a different type from the first data set (Para [0067], wherein a Multi-Layer Perceptron (MLP) model can be used as a gating model in a Mixture of Experts (ME) model in order to decide which base classifiers to use for a given input, as training set. The inputs to the gating model might include information about the quality of the different input data types, as the different types of the input data based on quality); combining the first AI model and the second AI model into an ensemble model (para [0074, wherein the process 200 may proceed to operation 206, wherein the system may ensemble the one or more gate classifiers); and providing the ensemble model to classify images (Para [0077] and [0055], wherein the process 200 may proceed to operation 208, wherein the system may apply the ensembled model to the automated environment. In some embodiment, the ensembled model may generate a semantic arrangement or a semantic map for the automated environment, and wherein the model types may comprise linear models, decision trees, neural networks. If the specific goal is classification (i.e., classifying data into one of several predefined categories)). Koh does not disclose room type from a room depicted in an image, to derive both the room type and an amenity found in the room, to derive a location from the image, or a combination thereof. Das discloses training models for deriving room type from a room depicted in an image (column 4, lines 1-3 and column 5, lines 5-7, wherein The LSTM network 140 is a recurrent neural network that has learned scene type data from corresponding training videos, and wherein exemplary scene types include kitchen, living room, dining room, family room, studio/room, office, full bathroom, half bathroom, bedroom, corridor, etc.). Koh and Das are combinable because they both disclose model data training in order to recognize objects from data. Therefore, before the effective filing data of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine the room type determination of Das’s system with Koh’s so that to provide assistance, advice, suggestions, or guidance to the user based on the type of object detected (column 1, lines 23-24).
Regarding claim 2, in the combination of Koh and Das, Koh further discloses the operations further comprising: combining, via a gating model, the first AI model and the second AI model into the ensemble model, wherein the gating model combines the first AI model and the second AI model by combining a first output contribution of the first AI model and a second output contribution of the second AI model into a final determination of the room type, the amenity, the location, or a combination thereof (Para [0065], wherein a typical architecture of ME models may comprise a set of base classifiers, a gating model, and a pooling model. Base classifiers are trained on a subset of the data for a specific task as described above, in order to specialize in certain types of tasks or patterns. A gating model decides what base classifiers' predictions should be considered in calculating the final output.).
Regarding claim 3, in the combination of Koh and Das, Koh further discloses wherein the gating model further combines the first output contribution of the first AI model and the second output contribution of the second AI model into the final determination based on an input test data set (para [0047], wherein next, the data may be slipped into training and test sets. Splitting the data into a training set and a test set may provide the benefit of evaluating the performance of the model. The training set may be used to train the model, while the test set may be used to evaluate the model's performance on unseen data).
Regarding claim 4, in the combination of Koh and Das, Koh further discloses wherein the gating model provides as input to the first AI model and to the second AI model the input test data set and to derive a first weight for the first AI model based on a first output of the first AI model and a second weight for the second AI model based on a second output of the second AI model, and wherein the first weight and the second weight are combined to derive the final determination (Para [0065], wherein a gating model decides what base classifiers' predictions should be considered in calculating the final output. The gating model is responsible for determining which base classifier is most appropriate for a given input, based on the characteristics of the input and the capabilities of the base classifiers. The pooling model combines the predictions of the selected base classifiers in a designated way (e.g., by taking a weighted average) in order to make the final prediction).
Regarding claim 5, in the combination of Koh and Das, Koh further discloses the operations further comprising: training a third AI model of the plurality of AI models to only derive the room type from the room depicted in the image, to derive both the room type and the amenity found in the room, to derive the location from the image, or a combination thereof, based on a third data set having a different type from the first data set and from the second data set; combining the first AI model, the second AI model, and the third AI model into the ensemble model; and providing the ensemble model to classify images (Para [0125], wherein an ensemble approach involves using a combination of multiple machine learning models, rather than relying on a single model. By using multiple models, the ensemble approach is able to capture a wider range of patterns in the data, which can improve the accuracy of the semantics extraction. Additionally, the ensemble approach allows for the use of different types of models for different types of data, which can further improve the accuracy of the semantics extraction, and wherein lack some criticality or unexpected results the exact number of MEs are within the skill level of the ordinary skill practitioner who would use the most appropriate number of AI models for a given application).
Regarding claim 6, in the combination of Koh and Das, Koh further discloses the operations further comprising: combining, via a gating model, the first AI model, the second AI model, and the third AI model into the ensemble model, wherein the gating model derives a final determination of the room type, the amenity, the location, or a combination thereof, based on a vote between the first AI model, the second AI model, and the third AI model (Para [0033], wherein the ensemble method may utilize a priority-based selector, inherently as voting. For example, to ensemble multiple base classifiers, a priority-based selector might be used to choose the base classifier(s) with the highest priority (e.g., the one that is most likely to produce the correct result) for a given input, and/or with a priority that is above a threshold).
Regarding claim 7, in the combination of Koh and Das, Koh further discloses wherein the gating model uses a weighted voting between the first AI model, the second AI model, and the third AI model to derive a final determination of the room type, the amenity, the location, or a combination thereof, based on a first weight assigned to the first AI model, a second weight assigned to the second AI model, and a third weight assigned to the third AI model (Para [0065], wherein a gating model decides what base classifiers' predictions should be considered in calculating the final output. The gating model is responsible for determining which base classifier is most appropriate for a given input, based on the characteristics of the input and the capabilities of the base classifiers. The pooling model combines the predictions of the selected base classifiers in a designated way (e.g., by taking a weighted average) in order to make the final prediction).
Regarding claim 8, in the combination of Koh and Das, Das further discloses wherein the operations further comprise training the first AI model of the plurality of AI models to only derive the room type and training the second AI model of the plurality of AI models to derive both the room type and the amenity found in the room based on the second data set having a different type from the first data set (column 11, lines 5-10 and column 6, lines 6-14, wherein a number of recent works has shown that CNN features trained on sufficiently large and diverse datasets, can be successfully transferred to other visual recognition tasks such as scene classification and object localization, as different models, with a only limited amount of task-specific training data, and wherein as shown in FIG. 3, the processor 132 is configured to determine scene type data from a frame sequence 176 of the image data generated by the camera 116 using the LSTM network 140, as the first data set in combination with the frame to determine the scene type. The processor 132 uses the object detector parameters 144, as the second data set to determine object as amenity in the room in combination with the frame, and hence different from the first data set combination, to generate data that identifies the likelihood that certain objects appear in the frame sequence 176 of imaged data processed by the processor 132).
Regarding claim 9, in the combination of Koh and Das, Koh further discloses wherein the first data set comprises a plurality of labeled images and wherein the second data set comprises a plurality of unlabeled images (Para [0020], wherein Example ML model types are gradient boosting, neural networks, and variants thereof. Some further examples of ML models may comprise: [0021] Supervised learning models: These models are trained on labeled data, which means that the data includes both input examples and corresponding correct output values. Examples of supervised learning models include decision trees, support vector machines, and logistic regression. [0022] Unsupervised learning models: These models are trained on unlabeled data).
Regarding claim 15, Koh discloses a non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations (Para [0088], wherein a memory unit storing data from inputs and program instructions to be executed by the processor unit) comprising: Please refer to the corresponding system claim 1 above for further teachings.
Regarding storage medium claims 16-17, please refer to the corresponding system claims 2 and 5 above for further teachings.
Regarding method claims 18-20, please refer to the corresponding system claims 2 and 5 above for further teachings.
Claims 12-14 are rejected under 35 U.S.C. 103 as being unpatentable over Koh in view of Das and further in view of US 2023/0274420 A1 to Seah et al (hereinafter ‘Seah’).
Regarding claim 12, Koh and Das do not specifically disclose wherein the first AI model comprises a transformer model architecture. Seah discloses wherein the first AI model comprises a transformer model architecture (Para [0009], wherein using a natural language processing component to generate a caption for the one or more medical images may comprise (i) using the transformer-based model to predict a probability for each word in the vocabulary and (ii) sampling one or more words using the probabilities from step (i).). Koh, Das Seah are combinable because they all disclose image object recognition process. Therefore, before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine the transformer model of Seah’s system with Koh’s and Das’, so that the image features are available without degradation at every step (para [0007]).
Regarding claim 13, in the combination of Koh, Das and Seah, Seah further discloses, wherein the transformer model architecture comprises: a multi-head attention layer comprising a plurality of heads each head focused on different portions of an input image (Para [0025], and Fig. 2, wherein the transformer-based model preferably comprises one or more encoder and zero or more decoder blocks, each comprising a multi-head attention layer), wherein the input image is converted into a plurality of input embeddings and positional embeddings to be processed via the multi-head attention layer (Para [0026], and Fig. 2, wherein the image processing component may be configured to produce an image feature tensor of size N×M, wherein M is the size of the embedding used by the transformer-based model and N is the number of images in the one or more images.); and a first add and normalize layer disposed downstream of the multi-head attention layer and configured to mitigate a vanishing gradient problem (Para [0095] and Fig. 2, wherein both the encoders and decoders also contain layer normalization steps 480a, 480b which normalize the weights to sum to 1. As illustrated on FIG. 2, each layer of the transformer (block) 420A, 420B has multiple attention heads, which capture the relevance of tokens to each other according to different definitions of relevance), wherein an output of the multi-head attention layer is provided as an input to the first add and normalize layer (para [0095] and Fig. 2, wherein Each block/layer 420A, 420B contains sublayers including: a linear multi-head attention layer 450, and a feedforward layer 460, separated by layer normalization steps 480a, 480b).
Regarding claim 14, in the combination of Koh, Das and Seah, Seah further discloses, wherein the transformer model architecture further comprises: a feed-forward layer disposed downstream of the multi-head attention layer and configured to apply a linear transformations to each input position of a feed-forward input (Fig 2, Feedforward 460); and a second add and normalize layer disposed downstream of the feed-forward layer and comprising a plurality of second heads each second head focused on different portions of an add and normalize input, wherein the first add and normalize layer provides the feed-forward layer with the feed-forward input and the feed-forward layer provides the second add and normalize layer with the add and normalize input (Para [0095], and Fig. 2, items 420B-470, wherein 420B has multiple attention heads, which capture the relevance of tokens to each other according to different definitions of relevance. As illustrated on FIG. 2, the model can contain multiple layers 420A, 420B, such as e.g. 12, each layer comprising an encoder or a decoder).
Allowable Subject Matter
Claims 10-11 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: the prior art or the prior art of record specifically, Koh, Das and US 12266160 B2 to He et al, does not disclose:
. . . . wherein the first data set comprises a first plurality of images divided into X grid divisions and the second data set comprises a second plurality of images divided into Y grid divisions, wherein X is not equal to Y, of claim 10 combined with other features and elements of the claim;
. . . wherein the first data set comprises a first plurality of images having a first image resolution and the second data set comprises a second plurality of images having a second image resolution different from the first image resolution, claim 11 combined with other features and elements of the claim.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHERVIN K NAKHJAVAN whose telephone number is (571)272-5731. The examiner can normally be reached Monday-Friday 9:00-12:00 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sue Lefkowitz can be reached at (571)272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHERVIN K NAKHJAVAN/Primary Examiner, Art Unit 2672
Prosecution Insights