Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The Amendment filed 09/23/2026 has been entered. Claims 1-20 remain pending in the application.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1,4, 8, 10, 11, 14, 18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over KAMATA (US 20230289594 A1) in view of Nosko et al. (US 20190050714 A1) hereinafter Nosko.
As to independent claim 1, Kamata teaches a method for controlling an artificial intelligence (AI) device, the method comprising:
obtaining, via a processor in the AI device, a plurality of universal modules; [device with M modules and layers Fig. 3, ¶25-28 " L layers, and each of the layers includes M neural network (NN) modules (Modules #1 to #M)"]
receiving, via the processor in the AI device, an input image and a query related to the input image; [question and image ¶33 "A question sentence and an image are input to each NN module in the first layer of the modular neural network."]
selecting, via the processor, a group of universal modules from among the plurality of universal modules; [selects modules in each layer (group) ¶130 "neural module processing unit 102 selects an NN module with the maximum weight (selected NN module) in each layer"]
outputting, via the processor, an answer based on the NMN, the query and the input image. [modular neural network with answer output ¶63 "output of each NN module in a final layer of the modular neural network, and obtains answer output as class classification from options"]
Kamata does not specifically teach determining, via the processor, a custom layout arrangement for the selected group of universal modules based on the query and dynamically connecting the selected group of universal modules together according to the custom layout arrangement to form a neural module network (NMN) tailored to the query.
However, Nosko teaches determining, via the processor, a custom layout arrangement for the selected group of universal modules based on the query and [selects modules and builds a network (layout) ¶19 "analyzing a task to be performed; deciding required properties of a dedicated neural network for performing the task; identifying suitable network modules according to the known training conditions; and linking the identified network modules to construct a the so dedicated network"] dynamically connecting the selected group of universal modules together according to the custom layout arrangement to form a neural module network (NMN) tailored to the query. [dynamically constructs neural network modules and selects a combo for an input task ¶35 " different modules are trained for different learning tasks, based on corresponding different datasets. Therefore, selection of specific combination of modules allows controlled and dynamic configuration of the resulting network for a specific task, by selecting from the combinatorial amount of possible different configurations and tasks."]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the modular model disclosed by Kamata by incorporating the determining, via the processor, a custom layout arrangement for the selected group of universal modules based on the query and dynamically connecting the selected group of universal modules together according to the custom layout arrangement to form a neural module network (NMN) tailored to the query disclosed by Nosko because both techniques address the same field of machine learning and by incorporating Nosko into Kamata allows more efficient application of algorithms and best instances of the networks are memorized [Nosko ¶5-6].
As to dependent claim 4, the rejection of claim 1 is incorporated, Kamata and Nosko further teach generating, via the processor, textural features or textual embeddings based on text of the query; and [Kamata word embeddings ¶61]
selecting, via the processor, the group of universal modules from among the plurality of universal modules based on the textural features or the textual embeddings. [Kamata selects modules in each layer (group) ¶130 "neural module processing unit 102 selects an NN module with the maximum weight (selected NN module) in each layer"]
As to dependent claim 8, the rejection of claim 1 is incorporated, Kamata and Nosko further teach wherein the group of universal modules are selected discretely or softly based on assigned weights. [Kamata selects based on max weight and token (discrete) ¶67]
As to dependent claim 10, the rejection of claim 1 is incorporated, Kamata and Nosko further teach wherein each of the plurality of universal modules is a neural network model. [Kamata M neural models as modules ¶28]
As to independent claim 11, Kamata teaches an artificial intelligence (AI) device, [device ¶24-25] the AI device comprising: a memory [memory ¶ 35] configured to store a plurality of universal modules; and a controller [processor ¶35] configured to: [device with M modules and layers Fig. 3, ¶25-28 " L layers, and each of the layers includes M neural network (NN) modules (Modules #1 to #M)"]
receive an input image and a query related to the input image, [question and image ¶33 "A question sentence and an image are input to each NN module in the first layer of the modular neural network."]
select a group of universal modules from among the plurality of universal modules, [selects modules in each layer (group) ¶130 "neural module processing unit 102 selects an NN module with the maximum weight (selected NN module) in each layer"]
outputting, via the processor, an answer based on the NMN, the query and the input image. [modular neural network with answer output ¶63 "output of each NN module in a final layer of the modular neural network, and obtains answer output as class classification from options"]
Kamata does not specifically teach determining, via the processor, a custom layout arrangement for the selected group of universal modules based on the query and dynamically connecting the selected group of universal modules together according to the custom layout arrangement to form a neural module network (NMN) tailored to the query.
However, Nosko teaches determining, via the processor, a custom layout arrangement for the selected group of universal modules based on the query and [selects modules and builds a network (layout) ¶19 "analyzing a task to be performed; deciding required properties of a dedicated neural network for performing the task; identifying suitable network modules according to the known training conditions; and linking the identified network modules to construct a the so dedicated network"] dynamically connecting the selected group of universal modules together according to the custom layout arrangement to form a neural module network (NMN) tailored to the query. [dynamically constructs neural network modules and selects a combo for an input task ¶35 " different modules are trained for different learning tasks, based on corresponding different datasets. Therefore, selection of specific combination of modules allows controlled and dynamic configuration of the resulting network for a specific task, by selecting from the combinatorial amount of possible different configurations and tasks."]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the modular model disclosed by Kamata by incorporating the determining, via the processor, a custom layout arrangement for the selected group of universal modules based on the query and dynamically connecting the selected group of universal modules together according to the custom layout arrangement to form a neural module network (NMN) tailored to the query disclosed by Nosko because both techniques address the same field of machine learning and by incorporating Nosko into Kamata allows more efficient application of algorithms and best instances of the networks are memorized [Nosko ¶5-6].
As to dependent claim 14, the rejection of claim 11 is incorporated, Kamata and Nosko further teach generating, via the processor, textural features or textual embeddings based on text of the query; and [Kamata word embeddings ¶61]
selecting, via the processor, the group of universal modules from among the plurality of universal modules based on the textural features or the textual embeddings. [Kamata selects modules in each layer (group) ¶130 "neural module processing unit 102 selects an NN module with the maximum weight (selected NN module) in each layer"]
As to dependent claim 18, the rejection of claim 11 is incorporated, Kamata and Nosko further teach wherein the group of universal modules are selected discretely or softly based on assigned weights. [Kamata selects based on max weight and token (discrete) ¶67]
As to dependent claim 20, the rejection of claim 11 is incorporated, Kamata and Nosko further teach wherein each of the plurality of universal modules is a neural network model. [Kamata M neural models as modules ¶28]
Claims 2-3 and 12-13 are rejected under 35 U.S.C. 103 as being unpatentable over Kamata and Nosko, as applied in the rejection of claims 1 and 11 above, and further in view of Ansari (US 20220188362 A1) hereinafter Ansari.
As to dependent claim 2, the rejection of claim 1 is incorporated.
Kamata and Nosko do not specifically teach wherein each of the plurality of universal modules corresponds to a single elemental sub-task.
However, Ansari teaches wherein each of the plurality of universal modules corresponds to a single elemental sub-task. [modules perform subtasks (A looks for Gray) in a filter task ¶30-33]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the modular model disclosed by Kamata and Nosko by incorporating the wherein each of the plurality of universal modules corresponds to a single elemental sub-task disclosed by Ansari because all techniques address the same field of machine learning and by incorporating Ansari into Kamata and Nosko improve outputs enabling more diverse answers for unseen concepts via collaboration [Ansari ¶3-4]
As to dependent claim 3, the rejection of claim 2 is incorporated, Kamata, Nosko and Ansari further teach wherein the single elemental sub-task includes one of a find task, a relocate task, an AND operation task, an OR operation task, a filter task, a count task, an exist task, a describe task, a less operation task, a more operation task, an equal operation task, and a compare task. [Ansari find, relate ¶2 filter ¶30-33, counting ¶24]
As to dependent claim 12, the rejection of claim 11 is incorporated.
Kamata and Nosko do not specifically teach wherein each of the plurality of universal modules corresponds to a single elemental sub-task.
However, Ansari teaches wherein each of the plurality of universal modules corresponds to a single elemental sub-task. [modules perform subtasks (A looks for Gray) in a filter task ¶30-33]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the modular model disclosed by Kamata and Nosko by incorporating the wherein each of the plurality of universal modules corresponds to a single elemental sub-task disclosed by Ansari because all techniques address the same field of machine learning and by incorporating Ansari into Kamata and Nosko improve outputs enabling more diverse answers for unseen concepts via collaboration [Ansari ¶3-4]
As to dependent claim 13, the rejection of claim 12 is incorporated, Kamata, Nosko and Ansari further teach wherein the single elemental sub-task includes one of a find task, a relocate task, an AND operation task, an OR operation task, a filter task, a count task, an exist task, a describe task, a less operation task, a more operation task, an equal operation task, and a compare task. [Ansari find, relate ¶2 filter ¶30-33, counting ¶24]
Claims 5-7 and 15-17 are rejected under 35 U.S.C. 103 as being unpatentable over Kamata and Nosko, as applied in the rejection of claims 1 and 11 above, and further in view of Chen et al. (US 20170124432 A1) hereinafter Chen.
As to dependent claim 5, the rejection of claim 1 is incorporated.
Kamata and Nosko do not specifically teach generating, via the processor, a feature map based on the input image; and determining, via the processor, the layout arrangement for the group of universal modules based on the feature map.
However, Chen teaches generating, via the processor, a feature map based on the input image; and [feature map for image ¶25-26]
determining, via the processor, the layout arrangement for the group of universal modules based on the feature map. [kernals (layout) based on map Fig. 5 515 ¶66 "In embodiments, the attention map generation part receives an image feature map extracted from an image input in step 505 and receives a dense question embeddings obtained from a question input using a LSTM in step 510. A set of configurable convolutional kernels are produced in step 515 by projecting the question embeddings from a semantic space into a visual space."]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the modular network model disclosed by Kamata and Nosko by incorporating the generating, via the processor, a feature map based on the input image; and determining, via the processor, the layout arrangement for the group of universal modules based on the feature map disclosed by Chen because all techniques address the same field of machine learning systems and by incorporating Chen into Kamata and Nosko provide more effective answers related to objects of interest [Chen ¶4, ¶48].
As to dependent claim 6, the rejection of claim 1 is incorporated.
Kamata and Nosko do not specifically teach wherein the plurality of universal modules includes at least two different types of visual modules configured to output a visual attention map, and at least two different types of classifier modules configured to output an answer.
However, Chen teaches wherein the plurality of universal modules includes at least two different types of visual modules configured to output a visual attention map, and at least two different types of classifier modules configured to output an answer. [question guided attention map and types (LSTM and CNN) of classifier ¶66 " generating a question-guided attention map using the ABC-CNN architecture, in accordance with embodiments of the present disclosure. In embodiments, the attention map generation part receives an image feature map extracted from an image input in step 505 and receives a dense question embeddings obtained from a question input using a LSTM in step 510"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the modular network model disclosed by Kamata and Nosko by incorporating the wherein the plurality of universal modules includes at least two different types of visual modules configured to output a visual attention map, and at least two different types of classifier modules configured to output an answer disclosed by Chen because all techniques address the same field of machine learning systems and by incorporating Chen into Kamata and Nosko provide more effective answers related to objects of interest [Chen ¶4, ¶48].
As to dependent claim 7, the rejection of claim 6 is incorporated, Kamata, Nosko and Chen further teach wherein the plurality of universal modules include: a first type of visual module configured to receive visual features and textual features, and output a visual attention map, [Chen image map and from image feature map and text (dense question embedding) ¶66]
a second type of visual module configured to receive visual features, an input visual attention map and textual features, and output a visual attention map, [Chen question guided map ¶66]
a first type of classifier module configured to receive visual features, an input visual attention map and textual features, and output a first answer, and [Chen multi-class classifier with lstm ¶65, ¶26 "a multi-class classifier is utilized to generate answers " ]
a second type of classifier module configured to receive visual features, a first input visual attention map, a second input visual attention map and textual features, and output a second answer. [Chen CNN Fig. 2 207 also has an arrow heading to answer ¶26]
As to dependent claim 15, the rejection of claim 11 is incorporated.
Kamata and Nosko do not specifically teach generate a feature map based on the input image, and determine the layout arrangement for the group of universal modules based on the feature map.
However, Chen teaches generate a feature map based on the input image, and [feature map for image ¶25-26]
determine the layout arrangement for the group of universal modules based on the feature map. [kernals (layout) based on map Fig. 5 515 ¶66 "In embodiments, the attention map generation part receives an image feature map extracted from an image input in step 505 and receives a dense question embeddings obtained from a question input using a LSTM in step 510. A set of configurable convolutional kernels are produced in step 515 by projecting the question embeddings from a semantic space into a visual space."]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the modular network model disclosed by Kamata and Nosko by incorporating the generating, via the processor, a feature map based on the input image; and determining, via the processor, the layout arrangement for the group of universal modules based on the feature map disclosed by Chen because all techniques address the same field of machine learning systems and by incorporating Chen into Kamata and Nosko provide more effective answers related to objects of interest [Chen ¶4, ¶48].
As to dependent claim 16, the rejection of claim 11 is incorporated.
Kamata and Nosko do not specifically teach wherein the plurality of universal modules includes at least two different types of visual modules configured to output a visual attention map, and at least two different types of classifier modules configured to output an answer.
However, Chen teaches wherein the plurality of universal modules includes at least two different types of visual modules configured to output a visual attention map, and at least two different types of classifier modules configured to output an answer. [question guided attention map and types (LSTM and CNN) of classifier ¶66 " generating a question-guided attention map using the ABC-CNN architecture, in accordance with embodiments of the present disclosure. In embodiments, the attention map generation part receives an image feature map extracted from an image input in step 505 and receives a dense question embeddings obtained from a question input using a LSTM in step 510"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the modular network model disclosed by Kamata and Nosko by incorporating the wherein the plurality of universal modules includes at least two different types of visual modules configured to output a visual attention map, and at least two different types of classifier modules configured to output an answer disclosed by Chen because all techniques address the same field of machine learning systems and by incorporating Chen into Kamata and Nosko provide more effective answers related to objects of interest [Chen ¶4, ¶48].
As to dependent claim 17, the rejection of claim 16 is incorporated, Kamata, Nosko and Chen further teach wherein the plurality of universal modules include: a first type of visual module configured to receive visual features and textual features, and output a visual attention map, [Chen image map and from image feature map and text (dense question embedding) ¶66]
a second type of visual module configured to receive visual features, an input visual attention map and textual features, and output a visual attention map, [Chen question guided map ¶66]
a first type of classifier module configured to receive visual features, an input visual attention map and textual features, and output a first answer, and [Chen multi-class classifier with lstm ¶65, ¶26 "a multi-class classifier is utilized to generate answers " ]
a second type of classifier module configured to receive visual features, a first input visual attention map, a second input visual attention map and textual features, and output a second answer. [Chen CNN Fig. 2 207 also has an arrow heading to answer ¶26]
Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Kamata and Nosko, as applied in the rejection of claims 1 and 11 above, and further in view of Amer et al. (US 20190303404 A1) hereinafter Amer.
As to dependent claim 9, the rejection of claim 1 is incorporated.
Kamata and Nosko further teach receiving, via the processor, a set of training samples and a number of epochs, each of the training samples including at least an image, a question and an answer; [Kamata trains repeatedly with epochs ¶96, training data with image/question/answer ¶32]
Kamata and Nosko do not specifically teach sorting, via the processor the training samples based on question length from shortest to longest to generate sorted training samples; and training the plurality of universal modules based on the sorted training samples and the number of epochs.
However, Amer teaches sorting, via the processor the training samples based on question length from shortest to longest to generate sorted training samples; and [sorts by length of sentence (shortest easiest first) ¶66, ¶69 " training samples may be divided into multiple difficulty groups based on sentence length (e.g., easy, medium, and hard difficulty sets). "
training the plurality of universal modules based on the sorted training samples and the number of epochs. [trains shortest easiest first ¶66, ¶69 ". Machine learning module 109 is first trained with the easiest set for 100 epochs before advancing to the medium and hard difficulty training sets, training for 100 epochs each"
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the modular network model disclosed by Kamata and Nosko by incorporating the sorting, via the processor the training samples based on question length from shortest to longest to generate sorted training samples; and training the plurality of universal modules based on the sorted training samples and the number of epochs disclosed by Amer because all techniques address the same field of machine learning systems and by incorporating Amer into Kamata and Nosko provide end users improved understanding of AI systems. [Amer ¶9]
As to dependent claim 19, the rejection of claim 11 is incorporated.
Kamata and Nosko further teach receiving, via the processor, a set of training samples and a number of epochs, each of the training samples including at least an image, a question and an answer; [Kamata trains repeatedly with epochs ¶96, training data with image/question/answer ¶32]
Kamata and Nosko do not specifically teach sorting, via the processor the training samples based on question length from shortest to longest to generate sorted training samples; and training the plurality of universal modules based on the sorted training samples and the number of epochs.
However, Amer teaches sorting, via the processor the training samples based on question length from shortest to longest to generate sorted training samples; and [sorts by length of sentence (shortest easiest first) ¶66, ¶69 " training samples may be divided into multiple difficulty groups based on sentence length (e.g., easy, medium, and hard difficulty sets). "
training the plurality of universal modules based on the sorted training samples and the number of epochs. [trains shortest easiest first ¶66, ¶69 ". Machine learning module 109 is first trained with the easiest set for 100 epochs before advancing to the medium and hard difficulty training sets, training for 100 epochs each"
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the modular network model disclosed by Kamata and Nosko by incorporating the sorting, via the processor the training samples based on question length from shortest to longest to generate sorted training samples; and training the plurality of universal modules based on the sorted training samples and the number of epochs disclosed by Amer because all techniques address the same field of machine learning systems and by incorporating Amer into Kamata and Nosko provide end users improved understanding of AI systems. [Amer ¶9]
Response to Arguments
Applicant's arguments filed 09/23/2026. In the remark, applicant argues that:
(1) Kamata and Ansari fail to teach new “determining, via the processor, a custom layout arrangement for the selected group of universal modules based on the query and dynamically connecting the selected group of universal modules together according to the custom layout arrangement to form a neural module network (NMN) tailored to the query; " as recited in amended claim 1 and 11.
As to point (1), Applicant’s arguments with respect to claim 1 have been considered but are moot in view of a new ground of rejection as set forth above of Kamata in view of Nosko. The concepts of "centralized assembly mechanism", "inventory of raw modules", "physically connects", "single cohesive network" and ""uniquely shaped" are not claimed but the remarks appear to discuss as if they are. Any prior art would not require these concepts. Note that Ansari constructs its network based on the question and image (see Fig. 3 304 and ¶37).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Applicant is required under 37 C.F.R. § 1.111(c) to consider these references fully when responding to this action.
Cheng et al. (US 20210232919 A1) teaches dynamic routing in a modular neural network (see ¶15)
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BEAU SPRATT whose telephone number is (571)272-9919. The examiner can normally be reached M-F 8:30-5 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached on 5712127212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BEAU D SPRATT/ Primary Examiner, Art Unit 2143
Prosecution Insights