Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-2, 5, 9-21 is/are rejected under 35 U.S.C. 103 as being obvious over R´ua et al (“MFAS: Multimodal Fusion Architecture Search” March 2019, cited in IDS of 12-6-2023) in view of So et al (“The Evolved Transformer” 2019)
The applied reference has a common inventor with the instant application. Based upon the earlier effectively filed date of the reference, it constitutes prior art under 35 U.S.C. 102(a)(2).
This rejection under 35 U.S.C. 103 might be overcome by: (1) a showing under 37 CFR 1.130(a) that the subject matter disclosed in the reference was obtained directly or indirectly from the inventor or a joint inventor of this application and is thus not prior art in accordance with 35 U.S.C.102(b)(2)(A); (2) a showing under 37 CFR 1.130(b) of a prior public disclosure under 35 U.S.C. 102(b)(2)(B); or (3) a statement pursuant to 35 U.S.C. 102(b)(2)(C) establishing that, not later than the effective filing date of the claimed invention, the subject matter disclosed and the claimed invention were either owned by the same person or subject to an obligation of assignment to the same person or subject to a joint research agreement. See generally MPEP § 717.02.
1. (Original) R´ua disclose a method comprising:
receiving training data for training a neural network to perform a multi-modal machine learning task, the training data comprising a plurality of training examples and a respective target output for each of the training examples (See section 3 on multimodal dataset and labels (i.e. respective target output for each of the training examples), wherein each training example comprises a respective network input from each of a plurality of modalities (See section 3 on x accounts for the first, y for the second one); and
PNG
media_image1.png
200
400
media_image1.png
Greyscale
PNG
media_image2.png
200
400
media_image2.png
Greyscale
determining, using the training data, an optimized neural network architecture for performing the machine learning task (See pg. 6970 and Algorithm 1 on return topK sampled archs), comprising:
PNG
media_image3.png
200
400
media_image3.png
Greyscale
PNG
media_image4.png
200
400
media_image4.png
Greyscale
generating a plurality of candidate neural network architectures task (See pg. 6970 and Algorithm 1 on sample K fusion architectures), wherein:
PNG
media_image5.png
200
400
media_image5.png
Greyscale
determining, for each candidate network architecture, a respective fitness on the multi-modal machine learning task (See pg. 6970 and Algorithm 1 on calculate accuracies (i.e. fitness)); and
PNG
media_image6.png
200
400
media_image6.png
Greyscale
selecting, from the candidate neural network architectures, the optimized neural network architecture based on the respective fitnesses (See pg. 6970 and Algorithm 1 on picking the absolute best based on validation accuracies).
PNG
media_image7.png
200
400
media_image7.png
Greyscale
While R´ua disclose neural architecture search, R´ua fails to disclose
“each candidate neural network architecture comprises a plurality of operation blocks that each receive a respective block input comprising at least a respective first input hidden state and a respective second hidden state and apply one or more operations to the block input to generate an output hidden state;
generating each of the plurality of candidate neural network architectures comprises, for each of the operation blocks in the candidate neural network architecture: selecting, for each of the plurality of operation blocks, each of the first and second hidden states that are received as input by the operation block from a respective set of possible inputs that includes at least each of the respective network inputs from each of the plurality of modalities;”
However, So disclose neural architecture search (thereby in the same field of endeavor) and apply NAS to Transformer architecture (See abstract), and further disclose
PNG
media_image8.png
200
400
media_image8.png
Greyscale
each candidate neural network architecture comprises a plurality of operation blocks that each receive a respective block input comprising at least a respective first input hidden state and a respective second hidden state and apply one or more operations to the block input to generate an output hidden state (See Fig. 1 and pg.3 on left and right hidden state input and new hidden state output);
generating each of the plurality of candidate neural network architectures comprises, for each of the operation blocks in the candidate neural network architecture: selecting, for each of the plurality of operation blocks, each of the first and second hidden states that are received as input by the operation block from a respective set of possible inputs that includes at least each of the respective network inputs from each of the plurality of modalities (See Fig. 1 and pg.3 on select hidden states as branch inputs).
PNG
media_image9.png
200
400
media_image9.png
Greyscale
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the NAS of R´ua to incorporate NAS of So.
Given the advantage of Evolved Transformer (So’s abstract “The architecture found in our experiments – the Evolved Trans former – demonstrates consistent improvement over the Transformer on four well-established language tasks”), one having ordinary skill in the art would have been motivated to make this obvious modification.
PNG
media_image10.png
200
400
media_image10.png
Greyscale
2. (Original) So disclose the method of any preceding claim, wherein the plurality of operation blocks are ordered in a sequence, and wherein, for each of the operation blocks, the respective set of possible inputs includes each of the respective network inputs from each of the plurality of modalities and the block outputs of any of the operation blocks that are before the operation block in the sequence (See Fig. 1 and pg.3 on subsequent blocks. Examiner Note: that implies the blocks are ordered in sequence).
PNG
media_image9.png
200
400
media_image9.png
Greyscale
5. (Currently Amended) R´ua disclose the method of wherein determining the fitness comprises: training a new neural network having the candidate neural network architecture on a training subset of the training data; and determining the fitness by evaluating a performance of the trained new neural network on a validation subset of the training data (See pg. 6970 on training and validation).
PNG
media_image11.png
200
400
media_image11.png
Greyscale
9. (Currently Amended) R´ua disclose the method of claim 1, wherein the plurality of modalities include audio data and corresponding video data and the multi-modal machine learning task is audio-video speech recognition (See pg 6966 on speech recognition) or visual question-answering (See pg. 6974 on visual question answering).
PNG
media_image12.png
200
400
media_image12.png
Greyscale
PNG
media_image13.png
200
400
media_image13.png
Greyscale
10. (Currently Amended) R´ua disclose the method of claim 1, wherein the plurality of modalities include content data and one or more of meta data for the content data or history data associated with a user being presented the content data, and wherein the multi-modal machine learning task is content recommendation (See pg. 6966 on variety of domains data and applications. Examiner Note: ¶ 1 applies. The content data and/or learning task is non functional descriptive material since the NAS is carried out in the same way regardless of the nature of the data/task)
PNG
media_image14.png
200
400
media_image14.png
Greyscale
11. (Currently Amended) ) So disclose the method of claim 1, wherein generating the plurality of candidate neural network architectures comprises: maintaining population data comprising, for each candidate neural network architecture in a population of candidate neural network architectures, data defining the candidate neural network architecture and the fitness for the candidate neural network architecture, and repeatedly performing the following operations: selecting a candidate neural network architecture from the population, generating a new candidate neural network architecture by applying one or more mutations to the selected candidate neural network architecture having the best fitness, determining a fitness for the new candidate neural network architecture, and adding the new candidate neural network architecture to the population (see pg. 2 on initialization, tournament selection, child production, mutation, fitness determination and repeat the process until population with high fitness individuals).
PNG
media_image15.png
200
400
media_image15.png
Greyscale
12. (Original) So disclose the method of claim 11, the following operations further comprising: selecting another plurality of candidate neural network architectures from the population, and removing from the population the candidate neural network architecture from the selected other plurality of candidate neural network architectures having a fitness that was determined least recently (see pg. 2 on tournament selection and killing/removing lowest fitness individuals).
PNG
media_image15.png
200
400
media_image15.png
Greyscale
.
13. (Currently Amended) So disclose the method of claim 11, wherein selecting, from the candidate neural network architectures, the optimized neural network architecture based on the respective fitnesses comprises, after repeatedly performing the operations: selecting the optimized neural network architecture from the population based on the fitnesses of the candidate neural network architectures in the population (see pg. 2 on tournament selection and repeating the process).
PNG
media_image16.png
200
400
media_image16.png
Greyscale
14. (Original) So disclose the method of claim 13, wherein the selecting the optimized neural network architecture comprises selecting the candidate neural network architecture in the population that has the best fitness (see pg. 2 on tournament selection and repeating the process until high fitness individuals are obtained).
PNG
media_image16.png
200
400
media_image16.png
Greyscale
15. (Original) So disclose the method of claim 13, wherein the selecting comprises: selecting, from the population, a plurality of candidate neural network architectures having the highest fitnesses; training respective neural networks having each of the plurality of selected candidate neural network architectures; determining a respective fitness for each of the trained respective neural networks; and selecting, as the optimized neural network architecture, the candidate neural network architecture corresponding to the trained respective neural network having the best fitness (see pg. 2 on tournament selection with highest fitness as parents and repeating the process until high fitness individuals are obtained).
PNG
media_image16.png
200
400
media_image16.png
Greyscale
16. (Currently Amended) R´ua disclose the method of claim 1, wherein determining, for each candidate network architecture, a respective fitness on the multi-modal machine learning task comprises: training a neural network having the candidate neural network architecture on at least a portion of the training data; and determining a fitness for the trained neural network based on a fitness measure for the multi-modal machine learning task (See pg. 6970 on training and validation set. Examiner Note: that means a portion of data is used in training and a portion is used in validation).
PNG
media_image17.png
200
400
media_image17.png
Greyscale
PNG
media_image18.png
200
400
media_image18.png
Greyscale
17. (Currently Amended) R´ua disclose the method of claim 1,further comprising providing data specifying the optimized architecture for use in performing the multi- modal machine learning task (See pg. 6970 on returning topK with corresponding architectures).
PNG
media_image19.png
200
400
media_image19.png
Greyscale
18. (Currently Amended) R´ua disclose the method of claim 1, further comprising: using a neural network having the optimized architecture to perform the multi-modal machine learning task on new examples (See pg. 6970 on applying the best one to obtain validation (i.e. new examples)).
PNG
media_image20.png
200
400
media_image20.png
Greyscale
19. (Currently Amended) So disclose the method of claim 1, wherein generating each of the plurality of candidate neural network architectures further comprises, for each of the operation blocks in the candidate neural network architecture: selecting respective values for each of one or more search fields that define the one or more operations that are applied by the operation block to the block input to the operation block (see pg. 3 on search space and operations of subsequent blocks).
PNG
media_image21.png
200
400
media_image21.png
Greyscale
Claims 20 and 21 are drawn to claim 1 and are rejected similarly. See pg. 6968 on computing power. Examiner Note: computing power indicated the presence of computing system with storage medium.
PNG
media_image22.png
200
400
media_image22.png
Greyscale
Claims 6-8 is/are rejected under 35 U.S.C. 103 as being obvious over R´ua et al (“MFAS: Multimodal Fusion Architecture Search” March 2019, cited in IDS of 12-6-2023) in view of So et al (“The Evolved Transformer” 2019) and further in view of Xue et al (“AWDF: An Adaptive Weighted Deep Fusion Architecture for Multi-modality Learning” 2019)
6. While R´ua disclose multimodal dataset, R´ua fails to apply the multimodal fusion disclose architecture search to EHR.
However, Xue disclose multi-modality learning and fusion architecture (thereby in the same field of endeavor) and apply it to EHR. Specifically, Xue further disclose
The method of claim 1, wherein each training example is data from an electronic health record and wherein the plurality of modalities comprise two or more of: contextual features (See pg. 2503 on patients’ information from MRI and demographics, which provide different context (intra-class and inter class), categorical features (See pg. 2503 on demographics), continuous features, or clinical notes (See pg. 2503 on diagnosis code and lab result).
PNG
media_image23.png
200
400
media_image23.png
Greyscale
PNG
media_image24.png
200
400
media_image24.png
Greyscale
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the multimodal application of NAS of R´ua to incorporate EHR of Xue.
Given the fact that EHR is one of the many multimodal area one can apply NAS to, and the benefit of using multimodal approach to EHR can provide a more comprehensive and holistic assessment (Xue’s introduction), one having ordinary skill in the art would have been motivated to make this obvious modification.
PNG
media_image25.png
200
400
media_image25.png
Greyscale
7. (Currently Amended) Xue disclose the method of claim 1 , wherein each training example is health data for a patient, and wherein the plurality of modalities includes one or more of: data of one or more modalities extracted from an electronic health record for the patient, medical image data for the patient (See pg. 2503 on image features), genomics data for the patient, or waveforms of speech or other audio data relevant to the patient.
PNG
media_image23.png
200
400
media_image23.png
Greyscale
8. (Currently Amended) Xue disclose the method of claim 6, wherein the multi-modal machine learning task is a task to predict an aspect of health of a patient from electronic health record data or health data for the patient (See pg. 2503 on early diagnosis)
PNG
media_image26.png
200
400
media_image26.png
Greyscale
Examiner’s Note
¶ 1: In re Curry, the Board held that in a computer-implemented method of providing "wellness-related services," "the 'wellness-related data in the databases.., does not functionally change either the data storage system or communication system used in the method of claim 81. Nonfunctional descriptive material cannot render nonobvious an invention that would have otherwise been obvious." See Ex parte Curry, 84 USPQ2d 1272 (BPAI 2005), aff'd (Fed. Cir. Appeal No. 2006-1003, aff'd Rule 36 June 12, 2006) MPEP 2106.01.
In re John, the Board held that the descriptive material (i.e., "control information" and "request" comprising a description of a development environment) recited in claim 1 is non-functional descriptive material because each of the "control information" and "request" does not functionally affect the process of managing a development environment. Rather, the control information is merely information that is used for "managing said first request" by a computer program and the request is data that is received ("receiving a first request") and processed ("processing said first request") by the system. In each case, the data (i.e., "control information" and "request") do not affect how the method of the prior art is performed on a computer system. In other words, the method of receiving and processing the request and reviewing the request "in accordance with control information" is carried out in the same way regardless of the nature of the request or control information. See Ex parte John F. Bisceglia, Appeal 2007-3447.
Allowable Subject Matter
Claims 3-4 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Claim 3: while So disclose 3 the method of claim 1,wherein: each candidate neural network architecture (i) generates an output state that is a combination of a respective output state for each of the plurality of modalities and (ii) processes at least the output state using one or more neural network layers to generate a final output for the multi-modal machine learning task (See Fig. 1 and pg.3 on combining block to produce a sing block output), none of the reference alone or in combination disclose “the respective output state for each of the plurality of modalities is generated from output hidden states that are both (i) not used in a block input for any of the plurality of operation blocks and (ii) conditioned on the respective network input from the modality”.
Pertinent Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Liang et al (US 2020/0143243 A1) disclose neural architecture search using evolutionary algorithm and multimodal. See [0065] and [0009].
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LUT WONG whose telephone number is (571)270-1123. The examiner can normally be reached M-F 10am-6pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abdullah Al Kawsar can be reached at 5712703169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LUT WONG/Primary Examiner, Art Unit 2127