Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1-20 are presented in the case.
Priority
Applicant's claim for the benefit of a prior-filed Provisional application 63/439,619 filed on 01/18/2023 is acknowledged.
Claim Objections
Claims 3-4 and 13-14 are objected to because of the following informalities:
Claim 3, line 1 recites the phrase “singe” which should be “single”
Claim 4, line 2 recites the phrase “textural” which should be “textual”
Claim 13, line 1 recites the phrase “singe” which should be “single”
Claim 14, line 2 recites the phrase “textural” which should be “textual”
For the informalities above and wherever else they may occur appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 8, 10, 11-14, 18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over KAMATA (US 20230289594 A1) in view of Ansari (US 20220188362 A1) hereinafter Ansari.
As to independent claim 1, Kamata teaches a method for controlling an artificial intelligence (AI) device, the method comprising:
obtaining, via a processor in the AI device, a plurality of universal modules; [device with M modules and layers Fig. 3, ¶25-28 " L layers, and each of the layers includes M neural network (NN) modules (Modules #1 to #M)"]
receiving, via the processor in the AI device, an input image and a query related to the input image; [question and image ¶33 "A question sentence and an image are input to each NN module in the first layer of the modular neural network."]
selecting, via the processor, a group of universal modules from among the plurality of universal modules; [selects modules in each layer (group) ¶130 "neural module processing unit 102 selects an NN module with the maximum weight (selected NN module) in each layer"]
outputting, via the processor, an answer based on the NMN, the query and the input image. [modular neural network with answer output ¶63 "output of each NN module in a final layer of the modular neural network, and obtains answer output as class classification from options"]
Kamata does not specifically teach determining, via the processor, a layout arrangement for the group of universal modules and connecting the group of universal modules together according to the layout arrangement to form a neural module network (NMN).
However, Ansari teaches determining, via the processor, a layout arrangement for the group of universal modules and connecting the group of universal modules together according to the layout arrangement to form a neural module network (NMN); [modular neural system with agents that make an program (layout) as a sequence of functions (modules) ¶21, ¶24 "agent can operate by reducing a question to one or more induced programs, i.e., a sequence of functions, each of which is a neural module. The induced programs can be subsequently executed to obtain an answer to the question."]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the modular model disclosed by Kamata by incorporating the determining, via the processor, a layout arrangement for the group of universal modules and connecting the group of universal modules together according to the layout arrangement to form a neural module network (NMN) disclosed by Ansari because both techniques address the same field of machine learning and by incorporating Ansari into Kamata improve outputs enabling more diverse answers for unseen concepts via collaboration [Ansari ¶3-4]
As to dependent claim 2, the rejection of claim 1 is incorporated, Kamata and Ansari further teach wherein each of the plurality of universal modules corresponds to a single elemental sub-task. [Ansari modules perform subtasks (A looks for Gray) in a filter task ¶30-33]
As to dependent claim 3, the rejection of claim 2 is incorporated, Kamata and Ansari further teach wherein the singe elemental sub-task includes one of a find task, a relocate task, an AND operation task, an OR operation task, a filter task, a count task, an exist task, a describe task, a less operation task, a more operation task, an equal operation task, and a compare task. [Ansari find, relate ¶2 filter ¶30-33, counting ¶24]
As to dependent claim 4, the rejection of claim 1 is incorporated, Kamata and Ansari further teach generating, via the processor, textural features or textual embeddings based on text of the query; and [Kamata word embeddings ¶61]
selecting, via the processor, the group of universal modules from among the plurality of universal modules based on the textural features or the textual embeddings. [Kamata selects modules in each layer (group) ¶130 "neural module processing unit 102 selects an NN module with the maximum weight (selected NN module) in each layer"]
As to dependent claim 8, the rejection of claim 1 is incorporated, Kamata and Ansari further teach wherein the group of universal modules are selected discretely or softly based on assigned weights. [Kamata selects based on max weight and token (discrete) ¶67]
As to dependent claim 10, the rejection of claim 1 is incorporated, Kamata and Ansari further teach wherein each of the plurality of universal modules is a neural network model. [Kamata M neural models as modules ¶28]
As to independent claim 11, Kamata teaches an artificial intelligence (AI) device, [device ¶24-25] the AI device comprising: a memory [memory ¶ 35] configured to store a plurality of universal modules; and a controller [processor ¶35] configured to: [device with M modules and layers Fig. 3, ¶25-28 " L layers, and each of the layers includes M neural network (NN) modules (Modules #1 to #M)"]
receive an input image and a query related to the input image, [question and image ¶33 "A question sentence and an image are input to each NN module in the first layer of the modular neural network."]
select a group of universal modules from among the plurality of universal modules, [selects modules in each layer (group) ¶130 "neural module processing unit 102 selects an NN module with the maximum weight (selected NN module) in each layer"]
outputting, via the processor, an answer based on the NMN, the query and the input image. [modular neural network with answer output ¶63 "output of each NN module in a final layer of the modular neural network, and obtains answer output as class classification from options"]
Kamata does not specifically teach determine a layout arrangement for the group of universal modules and connect the group of universal modules together according to the layout arrangement to form a neural module network (NMN).
However, Ansari teaches determine a layout arrangement for the group of universal modules and connect the group of universal modules together according to the layout arrangement to form a neural module network (NMN) [modular neural system with agents that make an program (layout) as a sequence of functions (modules) ¶21, ¶24 "agent can operate by reducing a question to one or more induced programs, i.e., a sequence of functions, each of which is a neural module. The induced programs can be subsequently executed to obtain an answer to the question."]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the modular model disclosed by Kamata by incorporating the determine a layout arrangement for the group of universal modules and connect the group of universal modules together according to the layout arrangement to form a neural module network (NMN) disclosed by Ansari because both techniques address the same field of machine learning and by incorporating Ansari into Kamata improve outputs enabling more diverse answers for unseen concepts via collaboration [Ansari ¶3-4]
As to dependent claim 12, the rejection of claim 11 is incorporated, Kamata and Ansari further teach wherein each of the plurality of universal modules corresponds to a single elemental sub-task. [Ansari modules perform subtasks (A looks for Gray) in a filter task ¶30-33]
As to dependent claim 13, the rejection of claim 12 is incorporated, Kamata and Ansari further teach wherein the singe elemental sub-task includes one of a find task, a relocate task, an AND operation task, an OR operation task, a filter task, a count task, an exist task, a describe task, a less operation task, a more operation task, an equal operation task, and a compare task. [Ansari find, relate ¶2 filter ¶30-33, counting ¶24]
As to dependent claim 14, the rejection of claim 11 is incorporated, Kamata and Ansari further teach generating, via the processor, textural features or textual embeddings based on text of the query; and [Kamata word embeddings ¶61]
selecting, via the processor, the group of universal modules from among the plurality of universal modules based on the textural features or the textual embeddings. [Kamata selects modules in each layer (group) ¶130 "neural module processing unit 102 selects an NN module with the maximum weight (selected NN module) in each layer"]
As to dependent claim 18, the rejection of claim 11 is incorporated, Kamata and Ansari further teach wherein the group of universal modules are selected discretely or softly based on assigned weights. [Kamata selects based on max weight and token (discrete) ¶67]
As to dependent claim 20, the rejection of claim 11 is incorporated, Kamata and Ansari further teach wherein each of the plurality of universal modules is a neural network model. [Kamata M neural models as modules ¶28]
Claims 5-7 and 15-17 are rejected under 35 U.S.C. 103 as being unpatentable over Kamata and Ansari, as applied in the rejection of claims 1 and 11 above, and further in view of Chen et al. (US 20170124432 A1) hereinafter Chen.
As to dependent claim 5, the rejection of claim 1 is incorporated.
Kamata and Ansari do not specifically teach generating, via the processor, a feature map based on the input image; and determining, via the processor, the layout arrangement for the group of universal modules based on the feature map.
However, Chen teaches generating, via the processor, a feature map based on the input image; and [feature map for image ¶25-26]
determining, via the processor, the layout arrangement for the group of universal modules based on the feature map. [kernals (layout) based on map Fig. 5 515 ¶66 "In embodiments, the attention map generation part receives an image feature map extracted from an image input in step 505 and receives a dense question embeddings obtained from a question input using a LSTM in step 510. A set of configurable convolutional kernels are produced in step 515 by projecting the question embeddings from a semantic space into a visual space."]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the modular network model disclosed by Kamata and Ansari by incorporating the generating, via the processor, a feature map based on the input image; and determining, via the processor, the layout arrangement for the group of universal modules based on the feature map disclosed by Chen because all techniques address the same field of machine learning systems and by incorporating Chen into Kamata and Ansari provide more effective answers related to objects of interest [Chen ¶4, ¶48].
As to dependent claim 6, the rejection of claim 1 is incorporated.
Kamata and Ansari do not specifically teach wherein the plurality of universal modules includes at least two different types of visual modules configured to output a visual attention map, and at least two different types of classifier modules configured to output an answer.
However, Chen teaches wherein the plurality of universal modules includes at least two different types of visual modules configured to output a visual attention map, and at least two different types of classifier modules configured to output an answer. [question guided attention map and types (LSTM and CNN) of classifier ¶66 " generating a question-guided attention map using the ABC-CNN architecture, in accordance with embodiments of the present disclosure. In embodiments, the attention map generation part receives an image feature map extracted from an image input in step 505 and receives a dense question embeddings obtained from a question input using a LSTM in step 510"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the modular network model disclosed by Kamata and Ansari by incorporating the wherein the plurality of universal modules includes at least two different types of visual modules configured to output a visual attention map, and at least two different types of classifier modules configured to output an answer disclosed by Chen because all techniques address the same field of machine learning systems and by incorporating Chen into Kamata and Ansari provide more effective answers related to objects of interest [Chen ¶4, ¶48].
As to dependent claim 7, the rejection of claim 6 is incorporated, Kamata, Ansari and Chen further teach wherein the plurality of universal modules include: a first type of visual module configured to receive visual features and textual features, and output a visual attention map, [Chen image map and from image feature map and text (dense question embedding) ¶66]
a second type of visual module configured to receive visual features, an input visual attention map and textual features, and output a visual attention map, [Chen question guided map ¶66]
a first type of classifier module configured to receive visual features, an input visual attention map and textual features, and output a first answer, and [Chen multi-class classifier with lstm ¶65, ¶26 "a multi-class classifier is utilized to generate answers " ]
a second type of classifier module configured to receive visual features, a first input visual attention map, a second input visual attention map and textual features, and output a second answer. [Chen CNN Fig. 2 207 also has an arrow heading to answer ¶26]
As to dependent claim 15, the rejection of claim 11 is incorporated.
Kamata and Ansari do not specifically teach generate a feature map based on the input image, and determine the layout arrangement for the group of universal modules based on the feature map.
However, Chen teaches generate a feature map based on the input image, and [feature map for image ¶25-26]
determine the layout arrangement for the group of universal modules based on the feature map. [kernals (layout) based on map Fig. 5 515 ¶66 "In embodiments, the attention map generation part receives an image feature map extracted from an image input in step 505 and receives a dense question embeddings obtained from a question input using a LSTM in step 510. A set of configurable convolutional kernels are produced in step 515 by projecting the question embeddings from a semantic space into a visual space."]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the modular network model disclosed by Kamata and Ansari by incorporating the generating, via the processor, a feature map based on the input image; and determining, via the processor, the layout arrangement for the group of universal modules based on the feature map disclosed by Chen because all techniques address the same field of machine learning systems and by incorporating Chen into Kamata and Ansari provide more effective answers related to objects of interest [Chen ¶4, ¶48].
As to dependent claim 16, the rejection of claim 11 is incorporated.
Kamata and Ansari do not specifically teach wherein the plurality of universal modules includes at least two different types of visual modules configured to output a visual attention map, and at least two different types of classifier modules configured to output an answer.
However, Chen teaches wherein the plurality of universal modules includes at least two different types of visual modules configured to output a visual attention map, and at least two different types of classifier modules configured to output an answer. [question guided attention map and types (LSTM and CNN) of classifier ¶66 " generating a question-guided attention map using the ABC-CNN architecture, in accordance with embodiments of the present disclosure. In embodiments, the attention map generation part receives an image feature map extracted from an image input in step 505 and receives a dense question embeddings obtained from a question input using a LSTM in step 510"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the modular network model disclosed by Kamata and Ansari by incorporating the wherein the plurality of universal modules includes at least two different types of visual modules configured to output a visual attention map, and at least two different types of classifier modules configured to output an answer disclosed by Chen because all techniques address the same field of machine learning systems and by incorporating Chen into Kamata and Ansari provide more effective answers related to objects of interest [Chen ¶4, ¶48].
As to dependent claim 17, the rejection of claim 16 is incorporated, Kamata, Ansari and Chen further teach wherein the plurality of universal modules include: a first type of visual module configured to receive visual features and textual features, and output a visual attention map, [Chen image map and from image feature map and text (dense question embedding) ¶66]
a second type of visual module configured to receive visual features, an input visual attention map and textual features, and output a visual attention map, [Chen question guided map ¶66]
a first type of classifier module configured to receive visual features, an input visual attention map and textual features, and output a first answer, and [Chen multi-class classifier with lstm ¶65, ¶26 "a multi-class classifier is utilized to generate answers " ]
a second type of classifier module configured to receive visual features, a first input visual attention map, a second input visual attention map and textual features, and output a second answer. [Chen CNN Fig. 2 207 also has an arrow heading to answer ¶26]
Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Kamata and Ansari, as applied in the rejection of claims 1 and 11 above, and further in view of Amer et al. (US 20190303404 A1) hereinafter Amer.
As to dependent claim 9, the rejection of claim 1 is incorporated.
Kamata and Ansar further teach receiving, via the processor, a set of training samples and a number of epochs, each of the training samples including at least an image, a question and an answer; [Kamata trains repeatedly with epochs ¶96, training data with image/question/answer ¶32]
Kamata and Ansari do not specifically teach sorting, via the processor the training samples based on question length from shortest to longest to generate sorted training samples; and training the plurality of universal modules based on the sorted training samples and the number of epochs.
However, Amer teaches sorting, via the processor the training samples based on question length from shortest to longest to generate sorted training samples; and [sorts by length of sentence (shortest easiest first) ¶66, ¶69 " training samples may be divided into multiple difficulty groups based on sentence length (e.g., easy, medium, and hard difficulty sets). "
training the plurality of universal modules based on the sorted training samples and the number of epochs. [trains shortest easiest first ¶66, ¶69 ". Machine learning module 109 is first trained with the easiest set for 100 epochs before advancing to the medium and hard difficulty training sets, training for 100 epochs each"
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the modular network model disclosed by Kamata and Ansari by incorporating the sorting, via the processor the training samples based on question length from shortest to longest to generate sorted training samples; and training the plurality of universal modules based on the sorted training samples and the number of epochs disclosed by Amer because all techniques address the same field of machine learning systems and by incorporating Amer into Kamata and Ansari provide end users improved understanding of AI systems. [Amer ¶9]
As to dependent claim 19, the rejection of claim 11 is incorporated.
Kamata and Ansar further teach receiving, via the processor, a set of training samples and a number of epochs, each of the training samples including at least an image, a question and an answer; [Kamata trains repeatedly with epochs ¶96, training data with image/question/answer ¶32]
Kamata and Ansari do not specifically teach sorting, via the processor the training samples based on question length from shortest to longest to generate sorted training samples; and training the plurality of universal modules based on the sorted training samples and the number of epochs.
However, Amer teaches sorting, via the processor the training samples based on question length from shortest to longest to generate sorted training samples; and [sorts by length of sentence (shortest easiest first) ¶66, ¶69 " training samples may be divided into multiple difficulty groups based on sentence length (e.g., easy, medium, and hard difficulty sets). "
training the plurality of universal modules based on the sorted training samples and the number of epochs. [trains shortest easiest first ¶66, ¶69 ". Machine learning module 109 is first trained with the easiest set for 100 epochs before advancing to the medium and hard difficulty training sets, training for 100 epochs each"
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the modular network model disclosed by Kamata and Ansari by incorporating the sorting, via the processor the training samples based on question length from shortest to longest to generate sorted training samples; and training the plurality of universal modules based on the sorted training samples and the number of epochs disclosed by Amer because all techniques address the same field of machine learning systems and by incorporating Amer into Kamata and Ansari provide end users improved understanding of AI systems. [Amer ¶9]
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Applicant is required under 37 C.F.R. § 1.111(c) to consider these references fully when responding to this action.
PERRY et al. (US 20220319096 A1) teaches machine learning of objects in a virtual environment including physics simulation (see ¶24).
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 U.S.P.Q. 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 U.S.P.Q. 275, 277 (C.C.P.A. 1968)).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Beau Spratt whose telephone number is 571 272 9919. The examiner can normally be reached 8:30am to 5:00pm (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached at 571 272 7212. The fax phone number for the organization where this application or proceeding is assigned is 571 483 7388.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866 217 9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800 786 9199 (IN USA OR CANADA) or 571 272 1000.
/BEAU D SPRATT/Primary Examiner, Art Unit 2143
Prosecution Insights