DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on XXXXXXXXXXXXXX has been entered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims X are canceled.
Claims X are amended.
Claims X are new.
Claims 1-20 are pending and have been examined.
This action is in reply to the papers filed on 06/04/2026 (effective filing date 08/03/2023).
Information Disclosure Statement
No Information Disclosure Statement has been filed.
The information disclosure statement(s) submitted: xxxxxxxx, has/have been considered by the Examiner and made of record in the application file.
Amendment
The present Office Action is based upon the original patent application filed on 08/03/2023 as modified by the amendment filed on 06/04/2026.
Reasons For Allowance
Prior-Art Rejection withdrawn
Claims xxx are allowed. The closest prior art (See PTO-892, Notice of References Cited) does not teach the claimed:
The closest prior-art (xxx) teach the features as disclosed in Non-final Rejection (xxxx), however, these cited references do not teach and the prior-art does not teach at least the following combination of features and/or elements:
determining, at a second time after associating the information corresponding to the first loyalty card with the logged location, that a second user computing device is located within a specified distance of the logged location using a second positioning system of the second user computing device; in response to determining that the second user computing device is located within the specified distance of the logged location of the first user computing device at the first time of detecting: retrieving information corresponding to a second loyalty card, the second loyalty card being associated with the merchant and the second user computing device; and displaying, by the second user computing device, data describing the second loyalty card.
Claim Rejections - 35 USC §101 - Withdrawn
Per Applicant’s amendments and arguments and considering new guidance in the MPEP, the rejections are withdrawn. Specifically, in Applicant’s Remarks (dated 03/14/2017, pgs. 8-11), Applicant traverses the 35 USC §101 rejections arguing that the amended claims recite new limitations that are not abstract, amount to significantly more, are directed to a practical application, etc… For example, Applicant argues….
In support of their arguments, Applicant cites to the following recent Fed. Cir. court cases (i.e., Alice Corp. v. CLS Bank Int’l, SRI Int’l, Inc. v. Cisco Systems, Inc., Ultramercial, Inc. v. Hulu, LLC, Berkheimer, Core Wireless, McRO, Enfish, Bascom, DDR, etc…).
Claim Rejections - 35 USC § 101
35 U.S.C. § 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. § 101 as being directed to non-statutory subject matter because the claimed invention is directed to an abstract idea without significantly more. These claims recite a method and system for implementing multi-model inferencing frameworks and application programming interfaces.
Claim 1 recites [a] method comprising: receiving, via a user application programming interface (API), configuration parameters for execution of a plurality of machine learning models (MLMs), wherein the configuration parameters identify: storage locations of the plurality of MLMs, storage locations of input data into the plurality of MLMs, and one or more inference applications for performing inference processing of the input data using the plurality of MLMs; configuring, using the received configuration parameters, the one or more inference applications to process the input data; executing, on one or more processing devices, the plurality of MLMs using the one or more inference applications to generate a plurality of sets of output data, wherein each MLM of the plurality of MLMs generates at least one set of output data of the plurality of sets of output data; and rendering, via the user API, a combined representation of the plurality of sets of output data.
The claims are being rejected according to the 2019 Revised Patent Subject Matter Eligibility Guidance (Federal Register, Vol. 84, No. 5, p. 50-57 (Jan. 7, 2019)).
Step 1: Does the Claim Fall within a Statutory Category?
Yes. Claims 1-10 recite a method and, therefore, are directed to the statutory class of a process. Claims 11-20 recite a system/processor and, therefore, are directed to the statutory class of machine.
Step 2A, Prong One: Is a Judicial Exception Recited?
Yes. The following tables identify the specific limitations that recite an abstract idea. The column that identifies the additional elements will be relevant to the analysis in step 2A, prong two, and step 2B.
Claim 1: Identification of Abstract Idea and Additional Elements, using Broadest Reasonable Interpretation
Claim Limitation
Abstract Idea
Additional Element
1. A method comprising:
No additional elements are positively claimed.
receiving, via a user application programming interface (API), configuration parameters for execution of a plurality of machine learning models (MLMs), wherein the configuration parameters identify: storage locations of the plurality of MLMs, storage locations of input data into the plurality of MLMs, and one or more inference applications for performing inference processing of the input data using the plurality of MLMs;
This limitation includes the step(s) of: receiving, via a user application programming interface (API), configuration parameters for execution of a plurality of machine learning models (MLMs), wherein the configuration parameters identify: storage locations of the plurality of MLMs, storage locations of input data into the plurality of MLMs, and one or more inference applications for performing inference processing of the input data using the plurality of MLMs.
But for the API and/or processing device, this limitation is directed to processing and/or communicating known information to implement multi-model inferencing frameworks and application programming interfaces which may be categorized as any of the following:
certain method of organizing human activity –
fundamental economic principles or practices (including hedging, insurance, mitigating risk), and/or
commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations).
receiving, via a user application programming interface (API), configuration parameters for execution of a plurality of machine learning models (MLMs)…
configuring, using the received configuration parameters, the one or more inference applications to process the input data;
This limitation includes the step(s) of: configuring, using the received configuration parameters, the one or more inference applications to process the input data.
No additional elements are positively claimed.
But for the API and/or processing device, this limitation is directed to processing and/or communicating known information to implement multi-model inferencing frameworks and application programming interfaces which may be categorized as any of the following:
certain method of organizing human activity –
fundamental economic principles or practices (including hedging, insurance, mitigating risk), and/or
commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations).
No additional elements are positively claimed.
executing, on one or more processing devices, the plurality of MLMs using the one or more inference applications to generate a plurality of sets of output data, wherein each MLM of the plurality of MLMs generates at least one set of output data of the plurality of sets of output data; and
This limitation includes the step(s) of: executing, on one or more processing devices, the plurality of MLMs using the one or more inference applications to generate a plurality of sets of output data, wherein each MLM of the plurality of MLMs generates at least one set of output data of the plurality of sets of output data.
But for the API and/or processing device, this limitation is directed to processing and/or communicating known information to implement multi-model inferencing frameworks and application programming interfaces which may be categorized as any of the following:
certain method of organizing human activity –
fundamental economic principles or practices (including hedging, insurance, mitigating risk), and/or
commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations).
executing, on one or more processing devices, the plurality of MLMs…
rendering, via the user API, a combined representation of the plurality of sets of output data.
This limitation includes the step(s) of: rendering, via the user API, a combined representation of the plurality of sets of output data.
But for the API and/or processing device, this limitation is directed to processing and/or communicating known information to implement multi-model inferencing frameworks and application programming interfaces which may be categorized as any of the following:
certain method of organizing human activity –
fundamental economic principles or practices (including hedging, insurance, mitigating risk), and/or
commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations).
rendering, via the user API, a combined representation…
As shown above, under Step 2A, Prong One, the claims recite a judicial exception (an abstract idea). The claims are directed to the abstract idea of implementing multi-model inferencing frameworks and application programming interfaces, which, pursuant to MPEP 2106.04, is aptly categorized as a method of organizing human activity. Therefore, under Step 2A, Prong One, the claims recite a judicial exception.
Next, the aforementioned claims recite additional functional elements that are associated with the judicial exception, including: an API for communicating information. Examiner understands these limitations to be insignificant extrasolution activity. (See Accenture, 728 F.3d 1336, 108 U.S.P.Q.2d 1173 (Fed. Cir. 2013), citing Cf. Diamond v. Diehr, 450 U.S. 175, 191-192 (1981) ("[I]nsignificant post-solution activity will not transform an unpatentable principle in to a patentable process.”).
The aforementioned claims also recite additional technical elements including: a “processor” or “processing device” to execute the method and system and an API for communicating data. These limitations are recited at a high level of generality and appear to be nothing more than generic computer components. Claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea eligible. Alice Corp., 134 S. Ct. at 2358, 110 USPQ2d at 1983. See also 134 S. Ct. at 2389, 110 USPQ2d at 1984.
Step 2A, Prong Two: Is the Abstract Idea Integrated into a Practical Application?
No. The judicial exception is not integrated into a practical application. The additional elements listed above that relate to computing components are recited at a high level of generality (i.e., as generic components performing generic computer functions such as communicating, receiving, processing, analyzing, and outputting/displaying data) such that they amount to no more than mere instructions to apply the exception using generic computing components. Simply implementing the abstract idea on a generic computer is not a practical application of the abstract idea. Additionally, the claims do not purport to improve the functioning of the computer itself. There is no technological problem that the claimed invention solves. Rather, the computer system is invoked merely as a tool. Accordingly, the additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. Therefore, these claims are directed to an abstract idea.
Furthermore, looking at the elements individually and in combination, under Step 2A, Prong Two, the claims as a whole do not integrate the judicial exception into a practical application because they fail to: improve the functioning of a computer or a technical field, apply the judicial exception in the treatment or prophylaxis of a disease, apply the judicial exception with a particular machine, effect a transformation or reduction of a particular article to a different state or thing, or apply the judicial exception beyond generally linking the use of the judicial exception to a particular technological environment. Rather, the claims merely use a computer as a tool to perform the abstract idea(s), and/or add insignificant extra-solution activity to the judicial exception, and/or generally link the use of the judicial exception to a particular technological environment.
Step 2B: Does the Claim Provide an Inventive Concept?
Next, under Step 2B, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements, when considered both individually and as an ordered combination, do not amount to significantly more than the abstract idea. Furthermore, looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. Simply put, as noted above, there is no indication that the combination of elements improves the functioning of a computer (or any other technology), and their collective functions merely provide conventional computer implementation. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements relating to computing components amount to no more than applying the exception using a generic computing components. Mere instructions to apply an exception using a generic computing component cannot provide an inventive concept. Furthermore, the broadest reasonable interpretation of the claimed computer components (i.e., additional elements) includes any generic computing components that are capable of being programmed to communicate, receive, send, process, analyze, output, or display data. Furthermore, Applicant’s Specification (PGPub. 2025/0045604 [0140]) refers to a general computer system, but they do not include any technically-specific computer algorithm or code.
Additionally, pursuant to the requirement under Berkheimer, the following citations are provided to demonstrate that the additional elements, identified as extra-solution activity, amount to activities that are well-understood, routine, and conventional. See MPEP 2106.05(d).
Capturing an image (code) with an RFID reader. Ritter, US Patent No. 7734507 (Col. 3, Lines 56-67); “RFID: Riding on the Chip” by Pat Russo. Frozen Food Age. New York: Dec. 2003, vol. 52, Issue 5; page S22.
Receiving or transmitting data over a network. Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362; OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014).
Storing and retrieving information in memory. Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93.
Outputting/Presenting data to a user. Mayo, 566 U.S. at 79, 101 USPQ2d at 1968; OIP Techs., Inc. v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1092-93 (Fed. Cir. 2015); MPEP 2106.05(g)(3).
Using a machine learning model to determine user segment characteristics for an ad campaign. https://whites.agency/blog/how-to-use-machine-learning-for-customer-segmentation/.
Thus, taken alone and in combination, the additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea), and are ineligible under 35 USC 101.
Independent system claim 11 and processor claim 20 also contains the identified abstract ideas, with the additional elements of a processor and storage medium, which are a generic computer components, and thus not significantly more for the same reasons and rationale above.
Dependent claims 2-10 and 12-19 further describe the abstract idea. The additional elements of the dependent claims fail to integrate the abstract idea into a practical application and do not amount to significantly more than the abstract idea. Thus, as the dependent claims remain directed to a judicial exception, and as the additional elements of the claims do not amount to significantly more, the dependent claims are not patent eligible.
As such, the claims are not patent eligible.
Therefore, the Office finds no improvements to another technology or field, no improvements to the function of the computer itself, and no meaningful limitations beyond generally linking the use of an abstract idea to a particular technological environment. Therefore, based on the two-part Alice Corp. analysis, there are no limitations in any of the claims that transform the exception (i.e., the abstract idea) into a patent eligible application.
Claim Rejections - Not an Ordered Combination
None of the limitations, considered as an ordered combination provide eligibility, because taken as a whole, the claims simply instruct the practitioner to implement the abstract idea with routine, conventional activity.
Claim Rejections - Preemption
Allowing the claims, as presently claimed, would preempt others from implementing multi-model inferencing frameworks and application programming interfaces. Furthermore, the claim language only recites the abstract idea of performing this method, there are no concrete steps articulating a particular way in which this idea is being implemented or describing how it is being performed.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over: Mallya et al. 2021/0142177; in view of Kuo et al. 2019/0156246.
18/229,929 – Claim 1. Mallya et al. 2021/0142177 teaches A method comprising: receiving, via a user application programming interface (API) (Mallya et al. 2021/0142177 [0079 - requests may be received through a user interface] In at least one embodiment, at a subsequent point in time, a request may be received from client device 602 (or another such device) for content (e.g., path determinations) or data that is at least partially determined or impacted by a trained neural network. This request can include, for example, input data to be processed using a neural network to obtain one or more inferences or other output values, classifications, or predictions. In at least one embodiment, input data can be received to interface layer 608 and directed to inference module 618, although a different system or service can be used as well. In at least one embodiment, inference module 618 can obtain an appropriate trained network, such as a trained deep neural network (DNN) as discussed herein, from model repository 616 if not already stored locally to inference module 618. Inference module 618 can provide data as input to a trained network, which can then generate one or more inferences as output. This may include, for example, a classification of an instance of input data. In at least one embodiment, inferences can then be transmitted to client device 602 for display or other communication to a user. In at least one embodiment, context data for a user may also be stored to a user context data repository 622, which may include data about a user which may be useful as input to a network in generating inferences, or determining data to return to a user after obtaining instances. In at least one embodiment, relevant data, which may include at least some of input or inference data, may also be stored to a local database 620 for processing future requests. In at least one embodiment, a user can use account or other information to access resources or functionality of a provider environment. In at least one embodiment, if permitted and available, user data may also be collected and used to further train models, in order to provide more accurate inferences for future requests. In at least one embodiment, requests may be received through a user interface to a machine learning application 626 executing on client device 602, and results displayed through a same interface. A client device can include resources such as a processor 628 and memory 630 for generating a request and processing results or a response, as well as at least one data storage element 632 for storing data for machine learning application 626. [0128 - graphical user interfaces] FIG. 12A is a block diagram illustrating an exemplary computer system, which may be a system with interconnected devices and components, a system-on-a-chip (SOC) or some combination thereof 1200 formed with a processor that may include execution units to execute an instruction, according to at least one embodiment. In at least one embodiment, computer system 1200 may include, without limitation, a component, such as a processor 1202 to employ execution units including logic to perform algorithms for process data, in accordance with present disclosure, such as in embodiment described herein. In at least one embodiment, computer system 1200 may include processors, such as PENTIUM® Processor family, Xeon™, Itanium®, XScale™ and/or StrongARM™, Intel® Core™, or Intel® Nervana™ microprocessors available from Intel Corporation of Santa Clara, Calif., although other systems (including PCs having other microprocessors, engineering workstations, set-top boxes and like) may also be used. In at least one embodiment, computer system 1200 may execute a version of WINDOWS' operating system available from Microsoft Corporation of Redmond, Wash., although other operating systems (UNIX and Linux for example), embedded software, and/or graphical user interfaces, may also be used. [0135 - user input and keyboard interfaces] In at least one embodiment, computer system 1200 may use system I/O 1222 that is a proprietary hub interface bus to couple MCH 1216 to I/O controller hub (“ICH”) 1230. In at least one embodiment, ICH 1230 may provide direct connections to some I/O devices via a local I/O bus. In at least one embodiment, local I/O bus may include, without limitation, a high-speed I/O bus for connecting peripherals to memory 1220, chipset, and processor 1202. Examples may include, without limitation, an audio controller 1229, a firmware hub (“flash BIOS”) 1228, a wireless transceiver 1226, a data storage 1224, a legacy I/O controller 1223 containing user input and keyboard interfaces 1225, a serial expansion port 1227, such as Universal Serial Bus (“USB”), and a network controller 1234. data storage 1224 may comprise a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or other mass storage device. [0393 - a request may be received by a set of API…] In at least one embodiment, shared storage may be mounted to AI services 4718 within system 4700. In at least one embodiment, shared storage may operate as a cache (or other storage device type) and may be used to process inference requests from applications. In at least one embodiment, when an inference request is submitted, a request may be received by a set of API instances of deployment system 4606, and one or more instances may be selected (e.g., for best fit, for load balancing, etc.) to process a request. In at least one embodiment, to process a request, a request may be entered into a database, a machine learning model may be located from model registry 4624 if not already in a cache, a validation step may ensure appropriate machine learning model is loaded into a cache (e.g., shared storage), and/or a copy of a model may be saved to a cache. In at least one embodiment, a scheduler (e.g., of pipeline manager 4712) may be used to launch an application that is referenced in a request if an application is not already running or if there are not enough instances of an application. In at least one embodiment, if an inference server is not already launched to execute a model, an inference server may be launched. Any number of inference servers may be launched per model. In at least one embodiment, in a pull model, in which inference servers are clustered, models may be cached whenever load balancing is advantageous. In at least one embodiment, inference servers may be statically loaded in corresponding, distributed servers.), configuration parameters (Mallya et al. 2021/0142177 [0121 - as shown in FIG. 11, framework layer 1120 includes a job scheduler 1122, a configuration manager 1124, a resource manager… configuration manager 1124 may be capable of configuring different layers such as software layer 1130 and framework layer… (interpreted as configuration parameters)][0126 - software and/or hardware resources described above may be configured as a service to allow users to train or performing inferencing of information… (interpreted as configuration parameters)][0100 - hyperparameter configurations] In at least one embodiment, instances of a dataset can be embedded into a lower dimensional space of a certain size during pre-processing. In at least one embodiment, a size of this space is a parameter to be tuned. In at least one embodiment, an architecture of a CNN contains many tunable parameters. A parameter for filter sizes can represent an interpretation of information that corresponds to a size of an instance that will be analyzed. In computational linguistics, this is known as an n-gram size. An example CNN uses three different filter sizes, which represent potentially different n-gram sizes. A number of filters per filter size can correspond to a depth of a filter. Each filter attempts to learn something different from a structure of an instance, such as a sentence structure for textual data. In a convolutional layer, an activation function can be a rectified linear unit and a pooling type set as max pooling. Results can then be concatenated into a single dimensional vector, and a last layer is fully connected onto a two-dimensional output. This corresponds to a binary classification to which an optimization function can be applied. One such function is an implementation of a Root Mean Square (RMS) propagation method of gradient descent, where example hyperparameters can include learning rate, batch size, maximum gradient normal, and epochs. With neural networks, regularization can be an extremely important consideration. In at least one embodiment input data may be relatively sparse. A main hyperparameter in such a situation can be a dropout at a penultimate layer, which represents a proportion of nodes that will not “fire” at each training cycle. An example training process can suggest different hyperparameter configurations based on feedback for a performance of previous configurations. This model can be trained with a proposed configuration, evaluated on a designated validation set, and performance reporting. This process can be repeated to, for example, trade off exploration (learning more about different configurations) and exploitation (leveraging previous knowledge to achieve better results). [0101 - configuration parameters] As training CNNs can be parallelized and GPU-enabled computing resources can be utilized, multiple optimization strategies can be attempted for different scenarios. A complex scenario allows tuning model architecture and preprocessing and stochastic gradient descent parameters. This expands a model configuration space. In a basic scenario, only preprocessing and stochastic gradient descent parameters are tuned. There can be a greater number of configuration parameters in a complex scenario than in a basic scenario. Tuning in a joint space can be performed using a linear or exponential number of steps, iteration through an optimization loop for models. A cost for such a tuning process can be significantly less than for tuning processes such as random search and grid search, without any significant performance loss.) for execution of a plurality of machine learning models (MLMs) (Mallya et al. 2021/0142177 [0334 - pipeline manager 4302 configures at least one of DPCs 4306 to implement a neural network model and/or a computing pipeline…][0376 – execute machine learning models] In at least one embodiment, where a service 4620 includes an AI service (e.g., an inference service), one or more machine learning models associated with an application for anomaly detection (e.g., tumors, growth abnormalities, scarring, etc.) may be executed by calling upon (e.g., as an API call) an inference service (e.g., an inference server) to execute machine learning model(s), or processing thereof, as part of application execution. In at least one embodiment, where another application includes one or more machine learning models for segmentation tasks, an application may call upon an inference service to execute machine learning models for performing one or more of processing operations associated with segmentation tasks. In at least one embodiment, software 4618 implementing advanced processing and inferencing pipeline that includes segmentation application and anomaly detection application may be streamlined because each application may call upon a same inference service to perform one or more inferencing tasks.[0383 - system 4700 may be configured to access and referenced data (e.g., DICOM data, RIS data, raw data, CIS data, REST compliant data, RPC data, raw data, etc.) from PACS servers (e.g., via a DICOM adapter 4702, or another data type adapter such as RIS, CIS, REST compliant, RPC, raw, etc.) to perform operations, such as training machine learning models, deploying machine learning models, image processing, inferencing, and/or other operations…][0376; 0392; 0398 – execute machine learning models]), wherein the configuration parameters identify (Mallya et al. 2021/0142177 [0101 - configuration parameters] As training CNNs can be parallelized and GPU-enabled computing resources can be utilized, multiple optimization strategies can be attempted for different scenarios. A complex scenario allows tuning model architecture and preprocessing and stochastic gradient descent parameters. This expands a model configuration space. In a basic scenario, only preprocessing and stochastic gradient descent parameters are tuned. There can be a greater number of configuration parameters in a complex scenario than in a basic scenario. Tuning in a joint space can be performed using a linear or exponential number of steps, iteration through an optimization loop for models. A cost for such a tuning process can be significantly less than for tuning processes such as random search and grid search, without any significant performance loss.): storage locations of the plurality of MLMs (Mallya et al. 2021/0142177 [0078 - a trained network can be stored to a model repository 616, for example, that may store different models or networks for users, applications, or services, etc. In at least one embodiment there may be multiple models for a single application or entity, as may be utilized based on a number of different factors…][0079 - data storage element 632 for storing data for machine learning application] In at least one embodiment, at a subsequent point in time, a request may be received from client device 602 (or another such device) for content (e.g., path determinations) or data that is at least partially determined or impacted by a trained neural network. This request can include, for example, input data to be processed using a neural network to obtain one or more inferences or other output values, classifications, or predictions. In at least one embodiment, input data can be received to interface layer 608 and directed to inference module 618, although a different system or service can be used as well. In at least one embodiment, inference module 618 can obtain an appropriate trained network, such as a trained deep neural network (DNN) as discussed herein, from model repository 616 if not already stored locally to inference module 618. Inference module 618 can provide data as input to a trained network, which can then generate one or more inferences as output. This may include, for example, a classification of an instance of input data. In at least one embodiment, inferences can then be transmitted to client device 602 for display or other communication to a user. In at least one embodiment, context data for a user may also be stored to a user context data repository 622, which may include data about a user which may be useful as input to a network in generating inferences, or determining data to return to a user after obtaining instances. In at least one embodiment, relevant data, which may include at least some of input or inference data, may also be stored to a local database 620 for processing future requests. In at least one embodiment, a user can use account or other information to access resources or functionality of a provider environment. In at least one embodiment, if permitted and available, user data may also be collected and used to further train models, in order to provide more accurate inferences for future requests. In at least one embodiment, requests may be received through a user interface to a machine learning application 626 executing on client device 602, and results displayed through a same interface. A client device can include resources such as a processor 628 and memory 630 for generating a request and processing results or a response, as well as at least one data storage element 632 for storing data for machine learning application 626. [0177 - memory location] In one embodiment, each WD 1984 is specific to a particular graphics acceleration module 1946 and/or graphics processing engines 1731-1732, N (shown in FIG. 17). It contains all information required by a graphics processing engine 1731-1732, N (shown in FIG. 17) to do work or it can be a pointer to a memory location where an application has set up a command queue of work to be completed.), storage locations of input data into the plurality of MLMs (Mallya et al. 2021/0142177 [0112; 0119][0368 - machine learning models may have been trained on imaging data from one location, two locations, or any number of locations] In at least one embodiment, training pipeline 4704 (FIG. 47) may include a scenario where facility 4602 needs a machine learning model for use in performing one or more processing tasks for one or more applications in deployment system 4606, but facility 4602 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, an existing machine learning model may be selected from a model registry 4624. In at least one embodiment, model registry 4624 may include machine learning models trained to perform a variety of different inference tasks on imaging data. In at least one embodiment, machine learning models in model registry 4624 may have been trained on imaging data from different facilities than facility 4602 (e.g., facilities remotely located). In at least one embodiment, machine learning models may have been trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, when being trained on imaging data from a specific location, training may take place at that location, or at least in a manner that protects confidentiality of imaging data or restricts imaging data from being transferred off-premises (e.g., to comply with HIPAA regulations, privacy regulations, etc.). In at least one embodiment, once a model is trained—or partially trained—at one location, a machine learning model may be added to model registry 4624. In at least one embodiment, a machine learning model may then be retrained, or updated, at any number of other facilities, and a retrained or updated model may be made available in model registry 4624. In at least one embodiment, a machine learning model may then be selected from model registry 4624—and referred to as output model 4616—and may be used in deployment system 4606 to perform one or more processing tasks for one or more applications of a deployment system. [0391 - data in same location of a memory may be used for any number of processing tasks] In at least one embodiment, services 4620 leveraged by and shared by applications or containers in deployment system 4606 may include compute services 4716, AI services 4718, visualization services 4720, and/or other service types. In at least one embodiment, applications may call (e.g., execute) one or more of services 4620 to perform processing operations for an application. In at least one embodiment, compute services 4716 may be leveraged by applications to perform super-computing or other high-performance computing (HPC) tasks. In at least one embodiment, compute service(s) 4716 may be leveraged to perform parallel processing (e.g., using a parallel computing platform 4730) for processing data through one or more of applications and/or one or more tasks of a single application, substantially simultaneously. In at least one embodiment, parallel computing platform 4730 (e.g., NVIDIA's CUDA) may enable general purpose computing on GPUs (GPGPU) (e.g., GPUs 4722). In at least one embodiment, a software layer of parallel computing platform 4730 may provide access to virtual instruction sets and parallel computational elements of GPUs, for execution of compute kernels. In at least one embodiment, parallel computing platform 4730 may include memory and, in some embodiments, a memory may be shared between and among multiple containers, and/or between and among different processing tasks within a single container. In at least one embodiment, inter-process communication (IPC) calls may be generated for multiple containers and/or for multiple processes within a container to use same data from a shared segment of memory of parallel computing platform 4730 (e.g., where multiple different stages of an application or multiple applications are processing same information). In at least one embodiment, rather than making a copy of data and moving data to different locations in memory (e.g., a read/write operation), same data in same location of a memory may be used for any number of processing tasks (e.g., at a same time, at different times, etc.). In at least one embodiment, as data is used to generate new data as a result of processing, this information of a new location of data may be stored and shared between various applications. In at least one embodiment, location of data and a location of updated or modified data may be part of a definition of how a payload is understood within containers.), and one or more inference applications (Mallya et al. 2021/0142177 [0396 - inference applications] In at least one embodiment, transfer of requests between services 4620 and inference applications may be hidden behind a software development kit (SDK), and robust transport may be provide through a queue. In at least one embodiment, a request will be placed in a queue via an API for an individual application/tenant ID combination and an SDK will pull a request from a queue and give a request to an application. In at least one embodiment, a name of a queue may be provided in an environment from where an SDK will pick it up. In at least one embodiment, asynchronous communication through a queue may be useful as it may allow any instance of an application to pick up work as it becomes available. Results may be transferred back through a queue, to ensure no data is lost. In at least one embodiment, queues may also provide an ability to segment work, as highest priority work may go to a queue with most instances of an application connected to it, while lowest priority work may go to a queue with a single instance connected to it that processes tasks in an order received. In at least one embodiment, an application may run on a GPU-accelerated instance generated in cloud 4726, and an inference service may perform inferencing on a GPU.) for performing inference processing (Mallya et al. 2021/0142177 [0073; 0079][0114 - inference processing] In at least one embodiment, activation storage 1020 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., Flash memory), or other storage. In at least one embodiment, activation storage 1020 may be completely or partially within or external to one or more processors or other logical circuits. In at least one embodiment, choice of whether activation storage 1020 is internal or external to a processor, for example, or comprised of DRAM, SRAM, Flash or some other storage type may depend on available storage on-chip versus off-chip, latency requirements of training and/or inferencing functions being performed, batch size of data used in inferencing and/or training of a neural network, or some combination of these factors. In at least one embodiment, inference and/or training logic 1015 illustrated in FIG. 9 may be used in conjunction with an application-specific integrated circuit (“ASIC”), such as Tensorflow® Processing Unit from Google, an inference processing unit (IPU) from Graphcore™, or a Nervana® (e.g., “Lake Crest”) processor from Intel Corp. In at least one embodiment, inference and/or training logic 1015 illustrated in FIG. 9 may be used in conjunction with central processing unit (“CPU”) hardware, graphics processing unit (“GPU”) hardware or other hardware, such as field programmable gate arrays (“FPGAs”). [0115 - inference processing] FIG. 10 illustrates inference and/or training logic 1015, according to at least one or more embodiments. In at least one embodiment, inference and/or training logic 1015 may include, without limitation, hardware logic in which computational resources are dedicated or otherwise exclusively used in conjunction with weight values or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, inference and/or training logic 1015 illustrated in FIG. 10 may be used in conjunction with an application-specific integrated circuit (ASIC), such as Tensorflow® Processing Unit from Google, an inference processing unit (IPU) from Graphcore™, or a Nervana® (e.g., “Lake Crest”) processor from Intel Corp. In at least one embodiment, inference and/or training logic 1015 illustrated in FIG. 10 may be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware or other hardware, such as field programmable gate arrays (FPGAs). In at least one embodiment, inference and/or training logic 1015 includes, without limitation, code and/or data storage 1001 and code and/or data storage 1005, which may be used to store code (e.g., graph code), weight values and/or other information, including bias values, gradient information, momentum values, and/or other parameter or hyperparameter information. In at least one embodiment illustrated in FIG. 10, each of code and/or data storage 1001 and code and/or data storage 1005 is associated with a dedicated computational resource, such as computational hardware 1002 and computational hardware 1006, respectively. In at least one embodiment, each of computational hardware 1002 and computational hardware 1006 comprises one or more ALUs that perform mathematical functions, such as linear algebraic functions, only on information stored in code and/or data storage 1001 and code and/or data storage 1005, respectively, result of which is stored in activation storage 1020.) of the input data using the plurality of MLMs (Mallya et al. 2021/0142177 [0079 – input data; 0085 – input data][0082; 0084; 0381][Fig. 9; 0012 - inference and/or training logic] FIG. 9 illustrates inference and/or training logic, according to at least one embodiment; [0013 - FIG. 10 illustrates inference and/or training logic] FIG. 10 illustrates inference and/or training logic, according to at least one embodiment; [0073 - Once a DNN is trained, this DNN can be deployed and used to identify and classify objects or patterns in a process known as inference. Examples of inference (a process through which a DNN extracts useful information from a given input)] A deep neural network (DNN) model includes multiple layers of many connected perceptrons (e.g., nodes) that can be trained with enormous amounts of input data to quickly solve complex problems with high accuracy. In one example, a first layer of a DNN model breaks down an input image of an automobile into various sections and looks for basic patterns such as lines and angles. Second layer assembles lines to look for higher-level patterns such as wheels, windshields, and mirrors. A next layer identifies a type of vehicle, and a final few layers generate a label for an input image, identifying a model of a specific automobile brand. Once a DNN is trained, this DNN can be deployed and used to identify and classify objects or patterns in a process known as inference. Examples of inference (a process through which a DNN extracts useful information from a given input) include identifying handwritten numbers on checks deposited into KIM machines, identifying images of friends in photos, delivering movie recommendations, identifying and classifying different types of automobiles, pedestrians, and road hazards in driverless cars, or translating human speech in near real-time. [0079 - input data to be processed using a neural network to obtain one or more inferences or other output values, classifications, or predictions. In at least one embodiment, input data can be received to interface layer 608 and directed to inference module] In at least one embodiment, at a subsequent point in time, a request may be received from client device 602 (or another such device) for content (e.g., path determinations) or data that is at least partially determined or impacted by a trained neural network. This request can include, for example, input data to be processed using a neural network to obtain one or more inferences or other output values, classifications, or predictions. In at least one embodiment, input data can be received to interface layer 608 and directed to inference module 618, although a different system or service can be used as well. In at least one embodiment, inference module 618 can obtain an appropriate trained network, such as a trained deep neural network (DNN) as discussed herein, from model repository 616 if not already stored locally to inference module 618. Inference module 618 can provide data as input to a trained network, which can then generate one or more inferences as output. This may include, for example, a classification of an instance of input data. In at least one embodiment, inferences can then be transmitted to client device 602 for display or other communication to a user. In at least one embodiment, context data for a user may also be stored to a user context data repository 622, which may include data about a user which may be useful as input to a network in generating inferences, or determining data to return to a user after obtaining instances. In at least one embodiment, relevant data, which may include at least some of input or inference data, may also be stored to a local database 620 for processing future requests. In at least one embodiment, a user can use account or other information to access resources or functionality of a provider environment. In at least one embodiment, if permitted and available, user data may also be collected and used to further train models, in order to provide more accurate inferences for future requests. In at least one embodiment, requests may be received through a user interface to a machine learning application 626 executing on client device 602, and results displayed through a same interface. A client device can include resources such as a processor 628 and memory 630 for generating a request and processing results or a response, as well as at least one data storage element 632 for storing data for machine learning application 626. [0107 - FIG. 9 illustrates inference and/or training logic 915 used to perform inferencing and/or training operations] FIG. 9 illustrates inference and/or training logic 915 used to perform inferencing and/or training operations associated with one or more embodiments. Details regarding inference and/or training logic 915 are provided below in conjunction with FIGS. 9 and/or 10.); configuring, using the received configuration parameters (Mallya et al. 2021/0142177 [0101 - configuration parameters] As training CNNs can be parallelized and GPU-enabled computing resources can be utilized, multiple optimization strategies can be attempted for different scenarios. A complex scenario allows tuning model architecture and preprocessing and stochastic gradient descent parameters. This expands a model configuration space. In a basic scenario, only preprocessing and stochastic gradient descent parameters are tuned. There can be a greater number of configuration parameters in a complex scenario than in a basic scenario. Tuning in a joint space can be performed using a linear or exponential number of steps, iteration through an optimization loop for models. A cost for such a tuning process can be significantly less than for tuning processes such as random search and grid search, without any significant performance loss.), the one or more inference applications (Mallya et al. 2021/0142177 [0396 - inference applications] In at least one embodiment, transfer of requests between services 4620 and inference applications may be hidden behind a software development kit (SDK), and robust transport may be provide through a queue. In at least one embodiment, a request will be placed in a queue via an API for an individual application/tenant ID combination and an SDK will pull a request from a queue and give a request to an application. In at least one embodiment, a name of a queue may be provided in an environment from where an SDK will pick it up. In at least one embodiment, asynchronous communication through a queue may be useful as it may allow any instance of an application to pick up work as it becomes available. Results may be transferred back through a queue, to ensure no data is lost. In at least one embodiment, queues may also provide an ability to segment work, as highest priority work may go to a queue with most instances of an application connected to it, while lowest priority work may go to a queue with a single instance connected to it that processes tasks in an order received. In at least one embodiment, an application may run on a GPU-accelerated instance generated in cloud 4726, and an inference service may perform inferencing on a GPU.) to process the input data (Mallya et al. 2021/0142177 [0108][0396 - inference applications…] In at least one embodiment, transfer of requests between services 4620 and inference applications may be hidden behind a software development kit (SDK), and robust transport may be provide through a queue. In at least one embodiment, a request will be placed in a queue via an API for an individual application/tenant ID combination and an SDK will pull a request from a queue and give a request to an application. In at least one embodiment, a name of a queue may be provided in an environment from where an SDK will pick it up. In at least one embodiment, asynchronous communication through a queue may be useful as it may allow any instance of an application to pick up work as it becomes available. Results may be transferred back through a queue, to ensure no data is lost. In at least one embodiment, queues may also provide an ability to segment work, as highest priority work may go to a queue with most instances of an application connected to it, while lowest priority work may go to a queue with a single instance connected to it that processes tasks in an order received. In at least one embodiment, an application may run on a GPU-accelerated instance generated in cloud 4726, and an inference service may perform inferencing on a GPU. [0079 - input data to be processed using a neural network to obtain one or more inferences or other output values, classifications, or predictions…] In at least one embodiment, at a subsequent point in time, a request may be received from client device 602 (or another such device) for content (e.g., path determinations) or data that is at least partially determined or impacted by a trained neural network. This request can include, for example, input data to be processed using a neural network to obtain one or more inferences or other output values, classifications, or predictions. In at least one embodiment, input data can be received to interface layer 608 and directed to inference module 618, although a different system or service can be used as well. In at least one embodiment, inference module 618 can obtain an appropriate trained network, such as a trained deep neural network (DNN) as discussed herein, from model repository 616 if not already stored locally to inference module 618. Inference module 618 can provide data as input to a trained network, which can then generate one or more inferences as output. This may include, for example, a classification of an instance of input data. In at least one embodiment, inferences can then be transmitted to client device 602 for display or other communication to a user. In at least one embodiment, context data for a user may also be stored to a user context data repository 622, which may include data about a user which may be useful as input to a network in generating inferences, or determining data to return to a user after obtaining instances. In at least one embodiment, relevant data, which may include at least some of input or inference data, may also be stored to a local database 620 for processing future requests. In at least one embodiment, a user can use account or other information to access resources or functionality of a provider environment. In at least one embodiment, if permitted and available, user data may also be collected and used to further train models, in order to provide more accurate inferences for future requests. In at least one embodiment, requests may be received through a user interface to a machine learning application 626 executing on client device 602, and results displayed through a same interface. A client device can include resources such as a processor 628 and memory 630 for generating a request and processing results or a response, as well as at least one data storage element 632 for storing data for machine learning application 626.); executing, on one or more processing devices, the plurality of MLMs (Mallya et al. 2021/0142177 execute machine learning models [0376; 0392; 0398]) using the one or more inference applications (Mallya et al. 2021/0142177 [0396 - inference applications] In at least one embodiment, transfer of requests between services 4620 and inference applications may be hidden behind a software development kit (SDK), and robust transport may be provide through a queue. In at least one embodiment, a request will be placed in a queue via an API for an individual application/tenant ID combination and an SDK will pull a request from a queue and give a request to an application. In at least one embodiment, a name of a queue may be provided in an environment from where an SDK will pick it up. In at least one embodiment, asynchronous communication through a queue may be useful as it may allow any instance of an application to pick up work as it becomes available. Results may be transferred back through a queue, to ensure no data is lost. In at least one embodiment, queues may also provide an ability to segment work, as highest priority work may go to a queue with most instances of an application connected to it, while lowest priority work may go to a queue with a single instance connected to it that processes tasks in an order received. In at least one embodiment, an application may run on a GPU-accelerated instance generated in cloud 4726, and an inference service may perform inferencing on a GPU.) to generate a plurality of sets of output data (Mallya et al. 2021/0142177 [0376 - execute machine learning model(s)] In at least one embodiment, where a service 4620 includes an AI service (e.g., an inference service), one or more machine learning models associated with an application for anomaly detection (e.g., tumors, growth abnormalities, scarring, etc.) may be executed by calling upon (e.g., as an API call) an inference service (e.g., an inference server) to execute machine learning model(s), or processing thereof, as part of application execution. In at least one embodiment, where another application includes one or more machine learning models for segmentation tasks, an application may call upon an inference service to execute machine learning models for performing one or more of processing operations associated with segmentation tasks. In at least one embodiment, software 4618 implementing advanced processing and inferencing pipeline that includes segmentation application and anomaly detection application may be streamlined because each application may call upon a same inference service to perform one or more inferencing tasks.), wherein each MLM of the plurality of MLMs generates at least one set of output data of the plurality of sets of output data (Mallya et al. 2021/0142177 [0079 - generate one or more inferences as output] In at least one embodiment, at a subsequent point in time, a request may be received from client device 602 (or another such device) for content (e.g., path determinations) or data that is at least partially determined or impacted by a trained neural network. This request can include, for example, input data to be processed using a neural network to obtain one or more inferences or other output values, classifications, or predictions. In at least one embodiment, input data can be received to interface layer 608 and directed to inference module 618, although a different system or service can be used as well. In at least one embodiment, inference module 618 can obtain an appropriate trained network, such as a trained deep neural network (DNN) as discussed herein, from model repository 616 if not already stored locally to inference module 618. Inference module 618 can provide data as input to a trained network, which can then generate one or more inferences as output. This may include, for example, a classification of an instance of input data. In at least one embodiment, inferences can then be transmitted to client device 602 for display or other communication to a user. In at least one embodiment, context data for a user may also be stored to a user context data repository 622, which may include data about a user which may be useful as input to a network in generating inferences, or determining data to return to a user after obtaining instances. In at least one embodiment, relevant data, which may include at least some of input or inference data, may also be stored to a local database 620 for processing future requests. In at least one embodiment, a user can use account or other information to access resources or functionality of a provider environment. In at least one embodiment, if permitted and available, user data may also be collected and used to further train models, in order to provide more accurate inferences for future requests. In at least one embodiment, requests may be received through a user interface to a machine learning application 626 executing on client device 602, and results displayed through a same interface. A client device can include resources such as a processor 628 and memory 630 for generating a request and processing results or a response, as well as at least one data storage element 632 for storing data for machine learning application 626.); and rendering, via the user API (Mallya et al. 2021/0142177 [0366 - an API may provide access to methods that allow users with appropriate credentials to associate models with applications] In at least one embodiment, model registry 4624 may be backed by object storage that may support versioning and object metadata. In at least one embodiment, object storage may be accessible through, for example, a cloud storage (e.g., cloud 4726 of FIG. 47) compatible application programming interface (API) from within a cloud platform. In at least one embodiment, machine learning models within model registry 4624 may uploaded, listed, modified, or deleted by developers or partners of a system interacting with an API. In at least one embodiment, an API may provide access to methods that allow users with appropriate credentials to associate models with applications, such that models may be executed as part of execution of containerized instantiations of applications.), a combined representation of the plurality of sets of output data (Mallya et al. 2021/0142177 [0383 - generated from computer models or renderings] In at least one embodiment, training pipelines 4704 may include AI-assisted annotation, as described in more detail herein with respect to at least FIG. 50B. In at least one embodiment, labeled clinic data 4612 (e.g., traditional annotation) may be generated by any number of techniques. In at least one embodiment, labels or other annotations may be generated within a drawing program (e.g., an annotation program), a computer aided design (CAD) program, a labeling program, another type of program suitable for generating annotations or labels for ground truth, and/or may be hand drawn, in some examples. In at least one embodiment, ground truth data may be synthetically produced (e.g., generated from computer models or renderings), real produced (e.g., designed and produced from real-world data), machine-automated (e.g., using feature analysis and learning to extract features from data and then generate labels), human annotated (e.g., labeler, or annotation expert, defines location of labels), and/or a combination thereof. In at least one embodiment, for each instance of imaging data 4608 (or other data type used by machine learning models), there may be corresponding ground truth data generated by training system 4604. In at least one embodiment, AI-assisted annotation may be performed as part of deployment pipelines 4710; either in addition to, or in lieu of AI-assisted annotation included in training pipelines 4704. In at least one embodiment, system 4700 may include a multi-layer platform that may include a software layer (e.g., software 4618) of diagnostic applications (or other application types) that may perform one or more medical imaging and diagnostic functions. In at least one embodiment, system 4700 may be communicatively coupled to (e.g., via encrypted links) PACS server networks of one or more facilities. In at least one embodiment, system 4700 may be configured to access and referenced data (e.g., DICOM data, RIS data, raw data, CIS data, REST compliant data, RPC data, raw data, etc.) from PACS servers (e.g., via a DICOM adapter 4702, or another data type adapter such as RIS, CIS, REST compliant, RPC, raw, etc.) to perform operations, such as training machine learning models, deploying machine learning models, image processing, inferencing, and/or other operations.).
Mallya et al. 2021/0142177 may not expressly disclose “configuring one or more inference applications” features, however, Kuo et al. 2019/0156246 teaches (Kuo et al. 2019/0156246 [0045 - different inference applications are configured to process data generated by different machine learning models] In some embodiments, a particular inference application is selected from among a group of different inference applications that are stored by the provider network (e.g., at a storage service). In embodiments, the different inference applications are configured to process data generated by different machine learning models.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Mallya et al. 2021/0142177 to include the features as taught by Kuo et al. 2019/0156246. One of ordinary skill in the art would have been motivated to do so to implement well known tools and features useful for implementing multi-model inferencing frameworks and application programming interfaces which should prove to improve user experience, maximize profits, and optimize revenue.
18/229,929 – Claim 11. Mallya et al. 2021/0142177 further teaches A system comprising: a memory device; and a processor, communicatively coupled to the memory device (Mallya et al. 2021/0142177 [0134][Fig. 11; 0118 - shown in FIG. 11, data center infrastructure layer 1110 may include … any number of central processing units (“CPUs”) or other processors … memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid state or disk drives), network input/output (“NW I/O”) devices, network…][0139 - system 1300 may include, without limitation, processor 1310 communicatively coupled to any suitable number or kind of components, peripherals, modules, or devices] In at least one embodiment, system 1300 may include, without limitation, processor 1310 communicatively coupled to any suitable number or kind of components, peripherals, modules, or devices. In at least one embodiment, processor 1310 coupled using a bus or interface, such as a 1° C. bus, a System Management Bus (“SMBus”), a Low Pin Count (LPC) bus, a Serial Peripheral Interface (“SPI”), a High Definition Audio (“HDA”) bus, a Serial Advance Technology Attachment (“SATA”) bus, a Universal Serial Bus (“USB”) (versions 1, 2, 3), or a Universal Asynchronous Receiver/Transmitter (“UART”) bus. In at least one embodiment, FIG. 13 illustrates a system, which includes interconnected hardware devices or “chips”, whereas in other embodiments, FIG. 13 may illustrate an exemplary System on a Chip (“SoC”). In at least one embodiment, devices illustrated in FIG. 13 may be interconnected with proprietary interconnects, standardized interconnects (e.g., PCIe) or some combination thereof In at least one embodiment, one or more components of FIG. 13 are interconnected using compute express link (CXL) interconnects. [0141; 0153; 0213; 0247]), to: …
receive, via a user application programming interface (API), configuration parameters for execution of a plurality of machine learning models (MLMs), wherein the configuration parameters identify: storage locations of the plurality of MLMs, storage locations of input data into the plurality of MLMs, and one or more inference applications for performing inference processing of the input data using the plurality of MLMs; configure, using the received configuration parameters, the one or more inference applications to process the input data; execute, on one or more processing devices, the plurality of MLMs using the one or more inference applications to generate a plurality of sets of output data, wherein each MLM of the plurality of MLMs generates at least one set of output data of the plurality of sets of output data; and render, via the user API, a combined representation of the plurality of sets of output data.
18/229,929 – Claim 20. Mallya et al. 2021/0142177 further teaches A processor comprising processing circuitry to perform operations comprising (Mallya et al. 2021/0142177 [0132 - general-purpose processor 1202, along with associated circuitry to execute instructions, operations used by…] In at least one embodiment, execution unit 1208, including, without limitation, logic to perform integer and floating point operations, also resides in processor 1202. In at least one embodiment, processor 1202 may also include a microcode (“ucode”) read only memory (“ROM”) that stores microcode for certain macro instructions. In at least one embodiment, execution unit 1208 may include logic to handle a packed instruction set 1209. In at least one embodiment, by including packed instruction set 1209 in an instruction set of a general-purpose processor 1202, along with associated circuitry to execute instructions, operations used by many multimedia applications may be performed using packed data in a general-purpose processor 1202. In one or more embodiments, many multimedia applications may be accelerated and executed more efficiently by using full width of a processor's data bus for performing operations on packed data, which may eliminate need to transfer smaller units of data across processor's data bus to perform one or more operations one data element at a time. [0216; 0247; 0264; 0270]): …
receive, via a user application programming interface (API), configuration parameters for execution of a plurality of machine learning models (MLMs), wherein the configuration parameters identify: storage locations of the plurality of MLMs, storage locations of input data into the plurality of MLMs, and one or more inference applications for performing inference processing of the input data using the plurality of MHLMs; configure, using the received configuration parameters, the one or more inference applications to process the input data; execute, on one or more processing devices, the plurality of MHLMs using the one or more inference applications to generate a plurality of sets of output data, wherein each MLM of the plurality of MLMs generates at least one set of output data of the plurality of sets of output data; and render, via the user API, a combined representation of the plurality of sets of output data.
Claims 11 and 20, have similar limitations as of Claim 1, therefore they are REJECTED under the same rationale as Claim 1.
18/229,929 – Claim 2. Mallya et al. 2021/0142177 further teaches The method of claim 1, wherein the configuration parameters (Mallya et al. 2021/0142177 [0101 - configuration parameters] As training CNNs can be parallelized and GPU-enabled computing resources can be utilized, multiple optimization strategies can be attempted for different scenarios. A complex scenario allows tuning model architecture and preprocessing and stochastic gradient descent parameters. This expands a model configuration space. In a basic scenario, only preprocessing and stochastic gradient descent parameters are tuned. There can be a greater number of configuration parameters in a complex scenario than in a basic scenario. Tuning in a joint space can be performed using a linear or exponential number of steps, iteration through an optimization loop for models. A cost for such a tuning process can be significantly less than for tuning processes such as random search and grid search, without any significant performance loss.) further identify: the one or more processing devices (Mallya et al. 2021/0142177 [0134][Fig. 11; 0118 - shown in FIG. 11, data center infrastructure layer 1110 may include … any number of central processing units (“CPUs”) or other processors … memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid state or disk drives), network input/output (“NW I/O”) devices, network…][0139 - system 1300 may include, without limitation, processor 1310 communicatively coupled to any suitable number or kind of components, peripherals, modules, or devices] In at least one embodiment, system 1300 may include, without limitation, processor 1310 communicatively coupled to any suitable number or kind of components, peripherals, modules, or devices. In at least one embodiment, processor 1310 coupled using a bus or interface, such as a 1° C. bus, a System Management Bus (“SMBus”), a Low Pin Count (LPC) bus, a Serial Peripheral Interface (“SPI”), a High Definition Audio (“HDA”) bus, a Serial Advance Technology Attachment (“SATA”) bus, a Universal Serial Bus (“USB”) (versions 1, 2, 3), or a Universal Asynchronous Receiver/Transmitter (“UART”) bus. In at least one embodiment, FIG. 13 illustrates a system, which includes interconnected hardware devices or “chips”, whereas in other embodiments, FIG. 13 may illustrate an exemplary System on a Chip (“SoC”). In at least one embodiment, devices illustrated in FIG. 13 may be interconnected with proprietary interconnects, standardized interconnects (e.g., PCIe) or some combination thereof In at least one embodiment, one or more components of FIG. 13 are interconnected using compute express link (CXL) interconnects. [0141; 0153; 0213; 0247]) for execution of the plurality of MLMs (Mallya et al. 2021/0142177 [0376 – execute machine learning models] In at least one embodiment, where a service 4620 includes an AI service (e.g., an inference service), one or more machine learning models associated with an application for anomaly detection (e.g., tumors, growth abnormalities, scarring, etc.) may be executed by calling upon (e.g., as an API call) an inference service (e.g., an inference server) to execute machine learning model(s), or processing thereof, as part of application execution. In at least one embodiment, where another application includes one or more machine learning models for segmentation tasks, an application may call upon an inference service to execute machine learning models for performing one or more of processing operations associated with segmentation tasks. In at least one embodiment, software 4618 implementing advanced processing and inferencing pipeline that includes segmentation application and anomaly detection application may be streamlined because each application may call upon a same inference service to perform one or more inferencing tasks.[0376; 0392; 0398 – execute machine learning models]), wherein the one or more processing devices comprise at least one of a central processing unit (CPU) or a graphics processing unit (GPU) (Mallya et al. 2021/0142177 [0080 – CPUs and GPUs] In at least one embodiment a processor 628 (or a processor of training manager 612 or inference module 618) will be a central processing unit (CPU). As mentioned, however, resources in such environments can utilize GPUs to process data for at least certain types of requests. With thousands of cores, GPUs are designed to handle substantial parallel workloads and, therefore, have become popular in deep learning for training neural networks and generating predictions. While use of GPUs for offline builds has enabled faster training of larger and more complex models, generating predictions offline implies that either request-time input features cannot be used or predictions must be generated for all permutations of features and stored in a lookup table to serve real-time requests. If a deep learning framework supports a CPU-mode and a model is small and simple enough to perform a feed-forward on a CPU with a reasonable latency, then a service on a CPU instance could host a model. In this case, training can be done offline on a GPU and inference done in real-time on a CPU. If a CPU approach is not viable, then a service can run on a GPU instance. Because GPUs have different performance and cost characteristics than CPUs, however, running a service that offloads a runtime algorithm to a GPU can require it to be designed differently from a CPU based service.[0114-0115]).
18/229,929 – Claim 12. The system of claim 11, wherein the configuration parameters further identify: the one or more processing devices for execution of the plurality of MLMs, wherein the one or more processing devices comprise at least one of a central processing unit (CPU) or a graphics processing unit (GPU).
Claim 12, has similar limitations as of Claim 2, therefore it is REJECTED under the same rationale as Claim 2.
18/229,929 – Claim 3. Mallya et al. 2021/0142177 further teaches The method of claim 1, wherein the configuration parameters further identify (Mallya et al. 2021/0142177 [0101 - configuration parameters] As training CNNs can be parallelized and GPU-enabled computing resources can be utilized, multiple optimization strategies can be attempted for different scenarios. A complex scenario allows tuning model architecture and preprocessing and stochastic gradient descent parameters. This expands a model configuration space. In a basic scenario, only preprocessing and stochastic gradient descent parameters are tuned. There can be a greater number of configuration parameters in a complex scenario than in a basic scenario. Tuning in a joint space can be performed using a linear or exponential number of steps, iteration through an optimization loop for models. A cost for such a tuning process can be significantly less than for tuning processes such as random search and grid search, without any significant performance loss.): a number format used for execution of one or more of the plurality of MLMs (Mallya et al. 2021/0142177 [0376 – execute machine learning models] In at least one embodiment, where a service 4620 includes an AI service (e.g., an inference service), one or more machine learning models associated with an application for anomaly detection (e.g., tumors, growth abnormalities, scarring, etc.) may be executed by calling upon (e.g., as an API call) an inference service (e.g., an inference server) to execute machine learning model(s), or processing thereof, as part of application execution. In at least one embodiment, where another application includes one or more machine learning models for segmentation tasks, an application may call upon an inference service to execute machine learning models for performing one or more of processing operations associated with segmentation tasks. In at least one embodiment, software 4618 implementing advanced processing and inferencing pipeline that includes segmentation application and anomaly detection application may be streamlined because each application may call upon a same inference service to perform one or more inferencing tasks.[0376; 0392; 0398 – execute machine learning models]), wherein the number format comprises at least one of an integer format, a single-precision format, or a double-precision format (Mallya et al. 2021/0142177 [0108 - integer and/or floating point units] In at least one embodiment, inference and/or training logic 915 may include, without limitation, code and/or data storage 901 to store forward and/or output weight and/or input/output data, and/or other parameters to configure neurons or layers of a neural network trained and/or used for inferencing in aspects of one or more embodiments. In at least one embodiment, training logic 915 may include, or be coupled to code and/or data storage 901 to store graph code or other software to control timing and/or order, in which weight and/or other parameter information is to be loaded to configure, logic, including integer and/or floating point units (collectively, arithmetic logic units (ALUs). In at least one embodiment, code, such as graph code, loads weight or other parameter information into processor ALUs based on an architecture of a neural network to which this code corresponds. In at least one embodiment, code and/or data storage 901 stores weight parameters and/or input/output data of each layer of a neural network trained or used in conjunction with one or more embodiments during forward propagation of input/output data and/or weight parameters during training and/or inferencing using aspects of one or more embodiments. In at least one embodiment, any portion of code and/or data storage 901 may be included with other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache or system memory. [0204 - perform single-precision (32-bit) and half-precision (16-bit) floating point operations, while DPFPUs 2515A-2515N perform double precision (64-bit) floating point operations] In at least one embodiment, FPUs 2514A-2514N can perform single-precision (32-bit) and half-precision (16-bit) floating point operations, while DPFPUs 2515A-2515N perform double precision (64-bit) floating point operations. In at least one embodiment, ALUs 2516A-2516N can perform variable precision integer operations at 8-bit, 16-bit, and 32-bit precision, and can be configured for mixed precision operations. In at least one embodiment, MPUs 2517A-2517N can also be configured for mixed precision matrix operations, including half-precision floating point and 8-bit integer operations. In at least one embodiment, MPUs 2517A-2517N can perform a variety of matrix operations to accelerate machine learning application frameworks, including enabling support for accelerated general matrix to matrix multiplication (GEMM). In at least one embodiment, AFUs 2512A-2512N can perform additional logic operations not supported by floating-point or integer units, including trigonometric operations (e.g., Sine, Cosine, etc.). [0244 - GPGPU cores 3162 can each include floating point units (FPUs) and/or integer arithmetic logic units (ALUs) that are used to execute instructions of graphics multiprocessor 3134. GPGPU cores 3162 can be similar in architecture or can differ in architecture. In at least one embodiment, a first portion of GPGPU cores 3162 include a single precision FPU and an integer ALU while a second portion of GPGPU cores include a double precision FPU] In at least one embodiment, GPGPU cores 3162 can each include floating point units (FPUs) and/or integer arithmetic logic units (ALUs) that are used to execute instructions of graphics multiprocessor 3134. GPGPU cores 3162 can be similar in architecture or can differ in architecture. In at least one embodiment, a first portion of GPGPU cores 3162 include a single precision FPU and an integer ALU while a second portion of GPGPU cores include a double precision FPU. In at least one embodiment, FPUs can implement IEEE 754-2008 standard for floating point arithmetic or enable variable precision floating point arithmetic. In at least one embodiment, graphics multiprocessor 3134 can additionally include one or more fixed function or special function units to perform specific functions such as copy rectangle or pixel blending operations. In at least one embodiment one or more of GPGPU cores can also include fixed or special function logic.).
18/229,929 – Claim 13. The system of claim 11, wherein the configuration parameters further identify: a number format used for execution of one or more of the plurality of MLMs, wherein the number format comprises at least one of an integer format, a single-precision format, or a double-precision format.
Claim 13, has similar limitations as of Claim 3, therefore it is REJECTED under the same rationale as Claim 3.
18/229,929 – Claim 4. Mallya et al. 2021/0142177 further teaches The method of claim 1, wherein the configuration parameters further identify (Mallya et al. 2021/0142177 [0101 - configuration parameters] As training CNNs can be parallelized and GPU-enabled computing resources can be utilized, multiple optimization strategies can be attempted for different scenarios. A complex scenario allows tuning model architecture and preprocessing and stochastic gradient descent parameters. This expands a model configuration space. In a basic scenario, only preprocessing and stochastic gradient descent parameters are tuned. There can be a greater number of configuration parameters in a complex scenario than in a basic scenario. Tuning in a joint space can be performed using a linear or exponential number of steps, iteration through an optimization loop for models. A cost for such a tuning process can be significantly less than for tuning processes such as random search and grid search, without any significant performance loss.) a type of execution of the plurality of MLMs (Mallya et al. 2021/0142177 [0376 – execute machine learning models] In at least one embodiment, where a service 4620 includes an AI service (e.g., an inference service), one or more machine learning models associated with an application for anomaly detection (e.g., tumors, growth abnormalities, scarring, etc.) may be executed by calling upon (e.g., as an API call) an inference service (e.g., an inference server) to execute machine learning model(s), or processing thereof, as part of application execution. In at least one embodiment, where another application includes one or more machine learning models for segmentation tasks, an application may call upon an inference service to execute machine learning models for performing one or more of processing operations associated with segmentation tasks. In at least one embodiment, software 4618 implementing advanced processing and inferencing pipeline that includes segmentation application and anomaly detection application may be streamlined because each application may call upon a same inference service to perform one or more inferencing tasks.[0376; 0392; 0398 – execute machine learning models]) on the one or more processing devices (Mallya et al. 2021/0142177 [0080 – CPUs and GPUs] In at least one embodiment a processor 628 (or a processor of training manager 612 or inference module 618) will be a central processing unit (CPU). As mentioned, however, resources in such environments can utilize GPUs to process data for at least certain types of requests. With thousands of cores, GPUs are designed to handle substantial parallel workloads and, therefore, have become popular in deep learning for training neural networks and generating predictions. While use of GPUs for offline builds has enabled faster training of larger and more complex models, generating predictions offline implies that either request-time input features cannot be used or predictions must be generated for all permutations of features and stored in a lookup table to serve real-time requests. If a deep learning framework supports a CPU-mode and a model is small and simple enough to perform a feed-forward on a CPU with a reasonable latency, then a service on a CPU instance could host a model. In this case, training can be done offline on a GPU and inference done in real-time on a CPU. If a CPU approach is not viable, then a service can run on a GPU instance. Because GPUs have different performance and cost characteristics than CPUs, however, running a service that offloads a runtime algorithm to a GPU can require it to be designed differently from a CPU based service.[0114-0115]), wherein the type of execution (Mallya et al. 2021/0142177 [0113 - distributed between different processors of different types (e.g., central processing units, graphics processing units, fixed function units, etc.); 0123] ) comprises at least one of: execution of a first MLM of the plurality of MLMs on a plurality of GPUs (Mallya et al. 2021/0142177 [0080 – CPUs and GPUs] In at least one embodiment a processor 628 (or a processor of training manager 612 or inference module 618) will be a central processing unit (CPU). As mentioned, however, resources in such environments can utilize GPUs to process data for at least certain types of requests. With thousands of cores, GPUs are designed to handle substantial parallel workloads and, therefore, have become popular in deep learning for training neural networks and generating predictions. While use of GPUs for offline builds has enabled faster training of larger and more complex models, generating predictions offline implies that either request-time input features cannot be used or predictions must be generated for all permutations of features and stored in a lookup table to serve real-time requests. If a deep learning framework supports a CPU-mode and a model is small and simple enough to perform a feed-forward on a CPU with a reasonable latency, then a service on a CPU instance could host a model. In this case, training can be done offline on a GPU and inference done in real-time on a CPU. If a CPU approach is not viable, then a service can run on a GPU instance. Because GPUs have different performance and cost characteristics than CPUs, however, running a service that offloads a runtime algorithm to a GPU can require it to be designed differently from a CPU based service.[0114-0115]), parallel execution of the first MLM and a second MLM of the plurality of MLMs on the one or more processing devices (Mallya et al. 2021/0142177 [0202 - FIGS. 25-26 illustrate additional exemplary graphics processor logic … FIG. 26 illustrates a highly-parallel general-purpose graphics processing unit…] FIGS. 25-26 illustrate additional exemplary graphics processor logic according to embodiments described herein. FIG. 25 illustrates a graphics core 2500 that may be included within graphics processor 2210 of FIG. 22, in at least one embodiment, and may be a unified shader core 2355A-2355N as in FIG. 24 in at least one embodiment. FIG. 26 illustrates a highly-parallel general-purpose graphics processing unit 2530 suitable for deployment on a multi-chip module in at least one embodiment. [0080 – CPUs and GPUs] In at least one embodiment a processor 628 (or a processor of training manager 612 or inference module 618) will be a central processing unit (CPU). As mentioned, however, resources in such environments can utilize GPUs to process data for at least certain types of requests. With thousands of cores, GPUs are designed to handle substantial parallel workloads and, therefore, have become popular in deep learning for training neural networks and generating predictions. While use of GPUs for offline builds has enabled faster training of larger and more complex models, generating predictions offline implies that either request-time input features cannot be used or predictions must be generated for all permutations of features and stored in a lookup table to serve real-time requests. If a deep learning framework supports a CPU-mode and a model is small and simple enough to perform a feed-forward on a CPU with a reasonable latency, then a service on a CPU instance could host a model. In this case, training can be done offline on a GPU and inference done in real-time on a CPU. If a CPU approach is not viable, then a service can run on a GPU instance. Because GPUs have different performance and cost characteristics than CPUs, however, running a service that offloads a runtime algorithm to a GPU can require it to be designed differently from a CPU based service.[0114-0115]), or sequential execution of the first MLM and the second MLM on the one or more processing devices (Mallya et al. 2021/0142177 [0385 - … include any number of applications that may be sequentially, non-sequentially, or otherwise applied to…] In at least one embodiment, deployment system 4606 may execute deployment pipelines 4710. In at least one embodiment, deployment pipelines 4710 may include any number of applications that may be sequentially, non-sequentially, or otherwise applied to imaging data (and/or other data types) generated by imaging devices, sequencing devices, genomics devices, etc.—including AI-assisted annotation, as described above. In at least one embodiment, as described herein, a deployment pipeline 4710 for an individual device may be referred to as a virtual instrument for a device (e.g., a virtual ultrasound instrument, a virtual CT scan instrument, a virtual sequencing instrument, etc.). In at least one embodiment, for a single device, there may be more than one deployment pipeline 4710 depending on information desired from data generated by a device. In at least one embodiment, where detections of anomalies are desired from an MRI machine, there may be a first deployment pipeline 4710, and where image enhancement is desired from output of an MRI machine, there may be a second deployment pipeline 4710. [0080 – CPUs and GPUs] In at least one embodiment a processor 628 (or a processor of training manager 612 or inference module 618) will be a central processing unit (CPU). As mentioned, however, resources in such environments can utilize GPUs to process data for at least certain types of requests. With thousands of cores, GPUs are designed to handle substantial parallel workloads and, therefore, have become popular in deep learning for training neural networks and generating predictions. While use of GPUs for offline builds has enabled faster training of larger and more complex models, generating predictions offline implies that either request-time input features cannot be used or predictions must be generated for all permutations of features and stored in a lookup table to serve real-time requests. If a deep learning framework supports a CPU-mode and a model is small and simple enough to perform a feed-forward on a CPU with a reasonable latency, then a service on a CPU instance could host a model. In this case, training can be done offline on a GPU and inference done in real-time on a CPU. If a CPU approach is not viable, then a service can run on a GPU instance. Because GPUs have different performance and cost characteristics than CPUs, however, running a service that offloads a runtime algorithm to a GPU can require it to be designed differently from a CPU based service.[0114-0115]).
18/229,929 – Claim 14. The system of claim 11, wherein the configuration parameters further identify a type of execution of the plurality of MLMs on the one or more processing devices, wherein the type of execution comprises at least one of: execution of a first MLM of the plurality of MLMs on a plurality of GPUs, parallel execution of the first MLM and a second MLM of the plurality of MLMs on the one or more processing devices, or sequential execution of the first MLM and the second MLM on the one or more processing devices.
Claim 14, has similar limitations as of Claim 4, therefore it is REJECTED under the same rationale as Claim 4.
18/229,929 – Claim 5. Mallya et al. 2021/0142177 further teaches The method of claim 1, wherein the configuration parameters (Mallya et al. 2021/0142177 [0101 - configuration parameters] As training CNNs can be parallelized and GPU-enabled computing resources can be utilized, multiple optimization strategies can be attempted for different scenarios. A complex scenario allows tuning model architecture and preprocessing and stochastic gradient descent parameters. This expands a model configuration space. In a basic scenario, only preprocessing and stochastic gradient descent parameters are tuned. There can be a greater number of configuration parameters in a complex scenario than in a basic scenario. Tuning in a joint space can be performed using a linear or exponential number of steps, iteration through an optimization loop for models. A cost for such a tuning process can be significantly less than for tuning processes such as random search and grid search, without any significant performance loss.) further identify at least one storage location for transient data (Mallya et al. 2021/0142177 [0368 - machine learning models may have been trained on imaging data from one location, two locations, or any number of locations] In at least one embodiment, training pipeline 4704 (FIG. 47) may include a scenario where facility 4602 needs a machine learning model for use in performing one or more processing tasks for one or more applications in deployment system 4606, but facility 4602 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, an existing machine learning model may be selected from a model registry 4624. In at least one embodiment, model registry 4624 may include machine learning models trained to perform a variety of different inference tasks on imaging data. In at least one embodiment, machine learning models in model registry 4624 may have been trained on imaging data from different facilities than facility 4602 (e.g., facilities remotely located). In at least one embodiment, machine learning models may have been trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, when being trained on imaging data from a specific location, training may take place at that location, or at least in a manner that protects confidentiality of imaging data or restricts imaging data from being transferred off-premises (e.g., to comply with HIPAA regulations, privacy regulations, etc.). In at least one embodiment, once a model is trained—or partially trained—at one location, a machine learning model may be added to model registry 4624. In at least one embodiment, a machine learning model may then be retrained, or updated, at any number of other facilities, and a retrained or updated model may be made available in model registry 4624. In at least one embodiment, a machine learning model may then be selected from model registry 4624—and referred to as output model 4616—and may be used in deployment system 4606 to perform one or more processing tasks for one or more applications of a deployment system. [0391 - data in same location of a memory may be used for any number of processing tasks] In at least one embodiment, services 4620 leveraged by and shared by applications or containers in deployment system 4606 may include compute services 4716, AI services 4718, visualization services 4720, and/or other service types. In at least one embodiment, applications may call (e.g., execute) one or more of services 4620 to perform processing operations for an application. In at least one embodiment, compute services 4716 may be leveraged by applications to perform super-computing or other high-performance computing (HPC) tasks. In at least one embodiment, compute service(s) 4716 may be leveraged to perform parallel processing (e.g., using a parallel computing platform 4730) for processing data through one or more of applications and/or one or more tasks of a single application, substantially simultaneously. In at least one embodiment, parallel computing platform 4730 (e.g., NVIDIA's CUDA) may enable general purpose computing on GPUs (GPGPU) (e.g., GPUs 4722). In at least one embodiment, a software layer of parallel computing platform 4730 may provide access to virtual instruction sets and parallel computational elements of GPUs, for execution of compute kernels. In at least one embodiment, parallel computing platform 4730 may include memory and, in some embodiments, a memory may be shared between and among multiple containers, and/or between and among different processing tasks within a single container. In at least one embodiment, inter-process communication (IPC) calls may be generated for multiple containers and/or for multiple processes within a container to use same data from a shared segment of memory of parallel computing platform 4730 (e.g., where multiple different stages of an application or multiple applications are processing same information). In at least one embodiment, rather than making a copy of data and moving data to different locations in memory (e.g., a read/write operation), same data in same location of a memory may be used for any number of processing tasks (e.g., at a same time, at different times, etc.). In at least one embodiment, as data is used to generate new data as a result of processing, this information of a new location of data may be stored and shared between various applications. In at least one embodiment, location of data and a location of updated or modified data may be part of a definition of how a payload is understood within containers.), wherein the transient data comprises at least one of: data output by one or more pre-processing operations that precede the inference processing, or data input into one or more post-processing operations that are subsequent to the inference processing (Mallya et al. 2021/0142177 [0371 - pre-processing as part of data processing pipeline to prepare data for processing by one or more applications. … post-processing may be performed on an output of one or more inferencing tasks or other processing tasks…] In at least one embodiment, a data processing pipeline may receive input data (e.g., imaging data 4608) in a DICOM, RIS, CIS, REST compliant, RPC, raw, and/or other format in response to an inference request (e.g., a request from a user of deployment system 4606, such as a clinician, a doctor, a radiologist, etc.). In at least one embodiment, input data may be representative of one or more images, video, and/or other data representations generated by one or more imaging devices, sequencing devices, radiology devices, genomics devices, and/or other device types. In at least one embodiment, data may undergo pre-processing as part of data processing pipeline to prepare data for processing by one or more applications. In at least one embodiment, post-processing may be performed on an output of one or more inferencing tasks or other processing tasks of a pipeline to prepare an output data for a next application and/or to prepare output data for transmission and/or use by a user (e.g., as a response to an inference request). In at least one embodiment, inferencing tasks may be performed by one or more machine learning models, such as trained or deployed neural networks, which may include output models 4616 of training system 4604. [0253 - post-processing and a multi-format…] In at least one embodiment, for at least some media processing commands, command streamer 3303 supplies commands to a video front end 3334, which couples with a media engine 3337. In at least one embodiment, media engine 3337 includes a Video Quality Engine (VQE) 3330 for video and image post-processing and a multi-format encode/decode (MFX) 3333 engine to provide hardware-accelerated media data encode and decode. In at least one embodiment, geometry pipeline 3336 and media engine 3337 each generate execution threads for thread execution resources provided by at least one graphics core 3380A. [0295 - pre-processing, and/or post-processing…] In at least one embodiment fixed, function block 3930 also includes a graphics SoC interface 3937, a graphics microcontroller 3938, and a media pipeline 3939. In at least one embodiment fixed, graphics SoC interface 3937 provides an interface between graphics core 3900 and other processor cores within a system on a chip integrated circuit. In at least one embodiment, graphics microcontroller 3938 is a programmable sub-processor that is configurable to manage various functions of graphics processor 3900, including thread dispatch, scheduling, and pre-emption. In at least one embodiment, media pipeline 3939 includes logic to facilitate decoding, encoding, pre-processing, and/or post-processing of multimedia data, including image and video data. In at least one embodiment, media pipeline 3939 implements media operations via requests to compute or sampling logic within sub-cores 3901-3901F.[0395; 0398; 0401]).
18/229,929 – Claim 15. The system of claim 11, wherein the configuration parameters further identify at least one storage location for transient data, wherein the transient data comprises at least one of: data output by one or more pre-processing operations that precede the inference processing, or data input into one or more post-processing operations that are subsequent to the inference processing.
Claim 15, has similar limitations as of Claim 5, therefore it is REJECTED under the same rationale as Claim 5.
18/229,929 – Claim 6. Mallya et al. 2021/0142177 further teaches The method of claim 5, wherein configuring the one or more inference applications (Mallya et al. 2021/0142177 [0396 - inference applications] In at least one embodiment, transfer of requests between services 4620 and inference applications may be hidden behind a software development kit (SDK), and robust transport may be provide through a queue. In at least one embodiment, a request will be placed in a queue via an API for an individual application/tenant ID combination and an SDK will pull a request from a queue and give a request to an application. In at least one embodiment, a name of a queue may be provided in an environment from where an SDK will pick it up. In at least one embodiment, asynchronous communication through a queue may be useful as it may allow any instance of an application to pick up work as it becomes available. Results may be transferred back through a queue, to ensure no data is lost. In at least one embodiment, queues may also provide an ability to segment work, as highest priority work may go to a queue with most instances of an application connected to it, while lowest priority work may go to a queue with a single instance connected to it that processes tasks in an order received. In at least one embodiment, an application may run on a GPU-accelerated instance generated in cloud 4726, and an inference service may perform inferencing on a GPU.) to process the input data comprises: causing reformatting of the transient data from a first format to a second format (Mallya et al. 2021/0142177 [0381 - format] In at least one embodiment, training system 4604 may execute training pipelines 4704, similar to those described herein with respect to FIG. 46. In at least one embodiment, where one or more machine learning models are to be used in deployment pipelines 4710 by deployment system 4606, training pipelines 4704 may be used to train or retrain one or more (e.g. pre-trained) models, and/or implement one or more of pre-trained models 4706 (e.g., without a need for retraining or updating). In at least one embodiment, as a result of training pipelines 4704, output model(s) 4616 may be generated. In at least one embodiment, training pipelines 4704 may include any number of processing steps, such as but not limited to imaging data (or other input data) conversion or adaption (e.g., using DICOM adapter 4702A to convert DICOM images to another format suitable for processing by respective machine learning models, such as Neuroimaging Informatics Technology Initiative (NIfTI) format), AI-assisted annotation 4610, labeling or annotating of imaging data 4608 to generate labeled clinic data 4612, model selection from a model registry, model training 4614, training, retraining, or updating models, and/or other processing steps. In at least one embodiment, for different machine learning models used by deployment system 4606, different training pipelines 4704 may be used. In at least one embodiment, training pipeline 4704 similar to a first example described with respect to FIG. 46 may be used for a first machine learning model, training pipeline 4704 similar to a second example described with respect to FIG. 46 may be used for a second machine learning model, and training pipeline 4704 similar to a third example described with respect to FIG. 46 may be used for a third machine learning model. In at least one embodiment, any combination of tasks within training system 4604 may be used depending on what is required for each respective machine learning model. In at least one embodiment, one or more of machine learning models may already be trained and ready for deployment so machine learning models may not undergo any processing by training system 4604, and may be implemented by deployment system 4606. [0404 - normalize or convert … file to format suitable for inference…] In at least one embodiment, CT reconstruction 4808 application and/or container may be executed once data (e.g., raw sinogram data) is available for processing by CT reconstruction 4808 application. In at least one embodiment, CT reconstruction 4808 may read raw sinogram data from a cache, reconstruct an image file out of raw sinogram data (e.g., as illustrated in visualization 4816B), and store resulting image file in a cache. In at least one embodiment, at completion of reconstruction, pipeline manager 4712 may be signaled that reconstruction task is complete. In at least one embodiment, once reconstruction is complete, and a reconstructed image file may be stored in a cache (or other storage device), organ segmentation 4810 application and/or container may be triggered by pipeline manager 4712. In at least one embodiment, organ segmentation 4810 application and/or container may read an image file from a cache, normalize or convert an image file to format suitable for inference (e.g., convert an image file to an input resolution of a machine learning model), and run inference against a normalized image. In at least one embodiment, to run inference on a normalized image, organ segmentation 4810 application and/or container may rely on services 4620, and pipeline manager 4712 and/or application orchestration system 4728 may facilitate use of services 4620 by organ segmentation 4810 application and/or container. For example, organ segmentation 4810 application and/or container may leverage AI services 4718 to perform inference on a normalized image, and AI services 4718 may leverage hardware 4622 (e.g., AI system 4724) to execute AI services 4718. In at least one embodiment, a result of an inference may be a mask file (e.g., as illustrated in visualization 4816C) that may be stored in a cache (or other storage device).), wherein at least one of the first format or the second format is a format compatible (Mallya et al. 2021/0142177 [0370 – compliant; 0373 – compliant with or compatible with…]) with the one or more inference applications (Mallya et al. 2021/0142177 [0396 - inference applications] In at least one embodiment, transfer of requests between services 4620 and inference applications may be hidden behind a software development kit (SDK), and robust transport may be provide through a queue. In at least one embodiment, a request will be placed in a queue via an API for an individual application/tenant ID combination and an SDK will pull a request from a queue and give a request to an application. In at least one embodiment, a name of a queue may be provided in an environment from where an SDK will pick it up. In at least one embodiment, asynchronous communication through a queue may be useful as it may allow any instance of an application to pick up work as it becomes available. Results may be transferred back through a queue, to ensure no data is lost. In at least one embodiment, queues may also provide an ability to segment work, as highest priority work may go to a queue with most instances of an application connected to it, while lowest priority work may go to a queue with a single instance connected to it that processes tasks in an order received. In at least one embodiment, an application may run on a GPU-accelerated instance generated in cloud 4726, and an inference service may perform inferencing on a GPU.).
18/229,929 – Claim 16. The system of claim 15, wherein to configure the one or more inference applications to process the input data, the processor is to: cause reformatting of the transient data from a first format to a second format, wherein at least one of the first format or the second format is a format accessible to the one or more inference applications.
Claim 16, has similar limitations as of Claim 6, therefore it is REJECTED under the same rationale as Claim 6.
18/229,929 – Claim 7. Mallya et al. 2021/0142177 further teaches The method of claim 1, wherein the configuration parameters (Mallya et al. 2021/0142177 [0101 - configuration parameters] As training CNNs can be parallelized and GPU-enabled computing resources can be utilized, multiple optimization strategies can be attempted for different scenarios. A complex scenario allows tuning model architecture and preprocessing and stochastic gradient descent parameters. This expands a model configuration space. In a basic scenario, only preprocessing and stochastic gradient descent parameters are tuned. There can be a greater number of configuration parameters in a complex scenario than in a basic scenario. Tuning in a joint space can be performed using a linear or exponential number of steps, iteration through an optimization loop for models. A cost for such a tuning process can be significantly less than for tuning processes such as random search and grid search, without any significant performance loss.) further identify at least one of: one or more pre-processing operations that precede the inference processing, or one or more post-processing operations that are subsequent to the inference processing (Mallya et al. 2021/0142177 [0371 - pre-processing as part of data processing pipeline to prepare data for processing by one or more applications. … post-processing may be performed on an output of one or more inferencing tasks or other processing tasks…] In at least one embodiment, a data processing pipeline may receive input data (e.g., imaging data 4608) in a DICOM, RIS, CIS, REST compliant, RPC, raw, and/or other format in response to an inference request (e.g., a request from a user of deployment system 4606, such as a clinician, a doctor, a radiologist, etc.). In at least one embodiment, input data may be representative of one or more images, video, and/or other data representations generated by one or more imaging devices, sequencing devices, radiology devices, genomics devices, and/or other device types. In at least one embodiment, data may undergo pre-processing as part of data processing pipeline to prepare data for processing by one or more applications. In at least one embodiment, post-processing may be performed on an output of one or more inferencing tasks or other processing tasks of a pipeline to prepare an output data for a next application and/or to prepare output data for transmission and/or use by a user (e.g., as a response to an inference request). In at least one embodiment, inferencing tasks may be performed by one or more machine learning models, such as trained or deployed neural networks, which may include output models 4616 of training system 4604. [0253 - post-processing and a multi-format…] In at least one embodiment, for at least some media processing commands, command streamer 3303 supplies commands to a video front end 3334, which couples with a media engine 3337. In at least one embodiment, media engine 3337 includes a Video Quality Engine (VQE) 3330 for video and image post-processing and a multi-format encode/decode (MFX) 3333 engine to provide hardware-accelerated media data encode and decode. In at least one embodiment, geometry pipeline 3336 and media engine 3337 each generate execution threads for thread execution resources provided by at least one graphics core 3380A. [0295 - pre-processing, and/or post-processing…] In at least one embodiment fixed, function block 3930 also includes a graphics SoC interface 3937, a graphics microcontroller 3938, and a media pipeline 3939. In at least one embodiment fixed, graphics SoC interface 3937 provides an interface between graphics core 3900 and other processor cores within a system on a chip integrated circuit. In at least one embodiment, graphics microcontroller 3938 is a programmable sub-processor that is configurable to manage various functions of graphics processor 3900, including thread dispatch, scheduling, and pre-emption. In at least one embodiment, media pipeline 3939 includes logic to facilitate decoding, encoding, pre-processing, and/or post-processing of multimedia data, including image and video data. In at least one embodiment, media pipeline 3939 implements media operations via requests to compute or sampling logic within sub-cores 3901-3901F.[0395; 0398; 0401]).
18/229,929 – Claim 8. Mallya et al. 2021/0142177 further teaches The method of claim 7, wherein executing the plurality of MLMs (Mallya et al. 2021/0142177 [0376 – execute machine learning models] In at least one embodiment, where a service 4620 includes an AI service (e.g., an inference service), one or more machine learning models associated with an application for anomaly detection (e.g., tumors, growth abnormalities, scarring, etc.) may be executed by calling upon (e.g., as an API call) an inference service (e.g., an inference server) to execute machine learning model(s), or processing thereof, as part of application execution. In at least one embodiment, where another application includes one or more machine learning models for segmentation tasks, an application may call upon an inference service to execute machine learning models for performing one or more of processing operations associated with segmentation tasks. In at least one embodiment, software 4618 implementing advanced processing and inferencing pipeline that includes segmentation application and anomaly detection application may be streamlined because each application may call upon a same inference service to perform one or more inferencing tasks.[0376; 0392; 0398 – execute machine learning models]) comprises performing: the one or more pre-processing operations that precede the inference processing, or the one or more post-processing operations that are subsequent to the inference processing (Mallya et al. 2021/0142177 [0371 - pre-processing as part of data processing pipeline to prepare data for processing by one or more applications. … post-processing may be performed on an output of one or more inferencing tasks or other processing tasks…] In at least one embodiment, a data processing pipeline may receive input data (e.g., imaging data 4608) in a DICOM, RIS, CIS, REST compliant, RPC, raw, and/or other format in response to an inference request (e.g., a request from a user of deployment system 4606, such as a clinician, a doctor, a radiologist, etc.). In at least one embodiment, input data may be representative of one or more images, video, and/or other data representations generated by one or more imaging devices, sequencing devices, radiology devices, genomics devices, and/or other device types. In at least one embodiment, data may undergo pre-processing as part of data processing pipeline to prepare data for processing by one or more applications. In at least one embodiment, post-processing may be performed on an output of one or more inferencing tasks or other processing tasks of a pipeline to prepare an output data for a next application and/or to prepare output data for transmission and/or use by a user (e.g., as a response to an inference request). In at least one embodiment, inferencing tasks may be performed by one or more machine learning models, such as trained or deployed neural networks, which may include output models 4616 of training system 4604. [0253 - post-processing and a multi-format…] In at least one embodiment, for at least some media processing commands, command streamer 3303 supplies commands to a video front end 3334, which couples with a media engine 3337. In at least one embodiment, media engine 3337 includes a Video Quality Engine (VQE) 3330 for video and image post-processing and a multi-format encode/decode (MFX) 3333 engine to provide hardware-accelerated media data encode and decode. In at least one embodiment, geometry pipeline 3336 and media engine 3337 each generate execution threads for thread execution resources provided by at least one graphics core 3380A. [0295 - pre-processing, and/or post-processing…] In at least one embodiment fixed, function block 3930 also includes a graphics SoC interface 3937, a graphics microcontroller 3938, and a media pipeline 3939. In at least one embodiment fixed, graphics SoC interface 3937 provides an interface between graphics core 3900 and other processor cores within a system on a chip integrated circuit. In at least one embodiment, graphics microcontroller 3938 is a programmable sub-processor that is configurable to manage various functions of graphics processor 3900, including thread dispatch, scheduling, and pre-emption. In at least one embodiment, media pipeline 3939 includes logic to facilitate decoding, encoding, pre-processing, and/or post-processing of multimedia data, including image and video data. In at least one embodiment, media pipeline 3939 implements media operations via requests to compute or sampling logic within sub-cores 3901-3901F.[0395; 0398; 0401]).
18/229,929 – Claim 9. Mallya et al. 2021/0142177 further teaches The method of claim 1, wherein the one or more inference applications (Mallya et al. 2021/0142177 [0396 - inference applications] In at least one embodiment, transfer of requests between services 4620 and inference applications may be hidden behind a software development kit (SDK), and robust transport may be provide through a queue. In at least one embodiment, a request will be placed in a queue via an API for an individual application/tenant ID combination and an SDK will pull a request from a queue and give a request to an application. In at least one embodiment, a name of a queue may be provided in an environment from where an SDK will pick it up. In at least one embodiment, asynchronous communication through a queue may be useful as it may allow any instance of an application to pick up work as it becomes available. Results may be transferred back through a queue, to ensure no data is lost. In at least one embodiment, queues may also provide an ability to segment work, as highest priority work may go to a queue with most instances of an application connected to it, while lowest priority work may go to a queue with a single instance connected to it that processes tasks in an order received. In at least one embodiment, an application may run on a GPU-accelerated instance generated in cloud 4726, and an inference service may perform inferencing on a GPU.) comprise at least one inference backend capable of being selectively directed to execute an MLM (Mallya et al. 2021/0142177 [0376 – execute machine learning models] In at least one embodiment, where a service 4620 includes an AI service (e.g., an inference service), one or more machine learning models associated with an application for anomaly detection (e.g., tumors, growth abnormalities, scarring, etc.) may be executed by calling upon (e.g., as an API call) an inference service (e.g., an inference server) to execute machine learning model(s), or processing thereof, as part of application execution. In at least one embodiment, where another application includes one or more machine learning models for segmentation tasks, an application may call upon an inference service to execute machine learning models for performing one or more of processing operations associated with segmentation tasks. In at least one embodiment, software 4618 implementing advanced processing and inferencing pipeline that includes segmentation application and anomaly detection application may be streamlined because each application may call upon a same inference service to perform one or more inferencing tasks.[0376; 0392; 0398 – execute machine learning models]) using a graphics processing unit (GPU) and/or a central processing unit (CPU) (Mallya et al. 2021/0142177 [0080 – CPUs and GPUs] In at least one embodiment a processor 628 (or a processor of training manager 612 or inference module 618) will be a central processing unit (CPU). As mentioned, however, resources in such environments can utilize GPUs to process data for at least certain types of requests. With thousands of cores, GPUs are designed to handle substantial parallel workloads and, therefore, have become popular in deep learning for training neural networks and generating predictions. While use of GPUs for offline builds has enabled faster training of larger and more complex models, generating predictions offline implies that either request-time input features cannot be used or predictions must be generated for all permutations of features and stored in a lookup table to serve real-time requests. If a deep learning framework supports a CPU-mode and a model is small and simple enough to perform a feed-forward on a CPU with a reasonable latency, then a service on a CPU instance could host a model. In this case, training can be done offline on a GPU and inference done in real-time on a CPU. If a CPU approach is not viable, then a service can run on a GPU instance. Because GPUs have different performance and cost characteristics than CPUs, however, running a service that offloads a runtime algorithm to a GPU can require it to be designed differently from a CPU based service.[0114-0115]) .
18/229,929 – Claim 10. Mallya et al. 2021/0142177 further teaches The method of claim 9, wherein the inference backend is further capable of being selectively directed to execute multiple MLMs (Mallya et al. 2021/0142177 [0376 – execute machine learning models] In at least one embodiment, where a service 4620 includes an AI service (e.g., an inference service), one or more machine learning models associated with an application for anomaly detection (e.g., tumors, growth abnormalities, scarring, etc.) may be executed by calling upon (e.g., as an API call) an inference service (e.g., an inference server) to execute machine learning model(s), or processing thereof, as part of application execution. In at least one embodiment, where another application includes one or more machine learning models for segmentation tasks, an application may call upon an inference service to execute machine learning models for performing one or more of processing operations associated with segmentation tasks. In at least one embodiment, software 4618 implementing advanced processing and inferencing pipeline that includes segmentation application and anomaly detection application may be streamlined because each application may call upon a same inference service to perform one or more inferencing tasks.[0376; 0392; 0398 – execute machine learning models]) sequentially (Mallya et al. 2021/0142177 [0385 - … include any number of applications that may be sequentially, non-sequentially, or otherwise applied to…] In at least one embodiment, deployment system 4606 may execute deployment pipelines 4710. In at least one embodiment, deployment pipelines 4710 may include any number of applications that may be sequentially, non-sequentially, or otherwise applied to imaging data (and/or other data types) generated by imaging devices, sequencing devices, genomics devices, etc.—including AI-assisted annotation, as described above. In at least one embodiment, as described herein, a deployment pipeline 4710 for an individual device may be referred to as a virtual instrument for a device (e.g., a virtual ultrasound instrument, a virtual CT scan instrument, a virtual sequencing instrument, etc.). In at least one embodiment, for a single device, there may be more than one deployment pipeline 4710 depending on information desired from data generated by a device. In at least one embodiment, where detections of anomalies are desired from an MRI machine, there may be a first deployment pipeline 4710, and where image enhancement is desired from output of an MRI machine, there may be a second deployment pipeline 4710. [0080 – CPUs and GPUs] In at least one embodiment a processor 628 (or a processor of training manager 612 or inference module 618) will be a central processing unit (CPU). As mentioned, however, resources in such environments can utilize GPUs to process data for at least certain types of requests. With thousands of cores, GPUs are designed to handle substantial parallel workloads and, therefore, have become popular in deep learning for training neural networks and generating predictions. While use of GPUs for offline builds has enabled faster training of larger and more complex models, generating predictions offline implies that either request-time input features cannot be used or predictions must be generated for all permutations of features and stored in a lookup table to serve real-time requests. If a deep learning framework supports a CPU-mode and a model is small and simple enough to perform a feed-forward on a CPU with a reasonable latency, then a service on a CPU instance could host a model. In this case, training can be done offline on a GPU and inference done in real-time on a CPU. If a CPU approach is not viable, then a service can run on a GPU instance. Because GPUs have different performance and cost characteristics than CPUs, however, running a service that offloads a runtime algorithm to a GPU can require it to be designed differently from a CPU based service.[0114-0115]) or in parallel (Mallya et al. 2021/0142177 [0202 - FIGS. 25-26 illustrate additional exemplary graphics processor logic … FIG. 26 illustrates a highly-parallel general-purpose graphics processing unit…] FIGS. 25-26 illustrate additional exemplary graphics processor logic according to embodiments described herein. FIG. 25 illustrates a graphics core 2500 that may be included within graphics processor 2210 of FIG. 22, in at least one embodiment, and may be a unified shader core 2355A-2355N as in FIG. 24 in at least one embodiment. FIG. 26 illustrates a highly-parallel general-purpose graphics processing unit 2530 suitable for deployment on a multi-chip module in at least one embodiment. [0080 – CPUs and GPUs] In at least one embodiment a processor 628 (or a processor of training manager 612 or inference module 618) will be a central processing unit (CPU). As mentioned, however, resources in such environments can utilize GPUs to process data for at least certain types of requests. With thousands of cores, GPUs are designed to handle substantial parallel workloads and, therefore, have become popular in deep learning for training neural networks and generating predictions. While use of GPUs for offline builds has enabled faster training of larger and more complex models, generating predictions offline implies that either request-time input features cannot be used or predictions must be generated for all permutations of features and stored in a lookup table to serve real-time requests. If a deep learning framework supports a CPU-mode and a model is small and simple enough to perform a feed-forward on a CPU with a reasonable latency, then a service on a CPU instance could host a model. In this case, training can be done offline on a GPU and inference done in real-time on a CPU. If a CPU approach is not viable, then a service can run on a GPU instance. Because GPUs have different performance and cost characteristics than CPUs, however, running a service that offloads a runtime algorithm to a GPU can require it to be designed differently from a CPU based service.[0114-0115]).
18/229,929 – Claim 17. Mallya et al. 2021/0142177 further teaches The system of claim 11, wherein the configuration parameters (Mallya et al. 2021/0142177 [0101 - configuration parameters] As training CNNs can be parallelized and GPU-enabled computing resources can be utilized, multiple optimization strategies can be attempted for different scenarios. A complex scenario allows tuning model architecture and preprocessing and stochastic gradient descent parameters. This expands a model configuration space. In a basic scenario, only preprocessing and stochastic gradient descent parameters are tuned. There can be a greater number of configuration parameters in a complex scenario than in a basic scenario. Tuning in a joint space can be performed using a linear or exponential number of steps, iteration through an optimization loop for models. A cost for such a tuning process can be significantly less than for tuning processes such as random search and grid search, without any significant performance loss.) further identify at least one of: one or more pre-processing operations that precede the inference processing, or one or more post-processing operations that are subsequent to the inference processing (Mallya et al. 2021/0142177 [0371 - pre-processing as part of data processing pipeline to prepare data for processing by one or more applications. … post-processing may be performed on an output of one or more inferencing tasks or other processing tasks…] In at least one embodiment, a data processing pipeline may receive input data (e.g., imaging data 4608) in a DICOM, RIS, CIS, REST compliant, RPC, raw, and/or other format in response to an inference request (e.g., a request from a user of deployment system 4606, such as a clinician, a doctor, a radiologist, etc.). In at least one embodiment, input data may be representative of one or more images, video, and/or other data representations generated by one or more imaging devices, sequencing devices, radiology devices, genomics devices, and/or other device types. In at least one embodiment, data may undergo pre-processing as part of data processing pipeline to prepare data for processing by one or more applications. In at least one embodiment, post-processing may be performed on an output of one or more inferencing tasks or other processing tasks of a pipeline to prepare an output data for a next application and/or to prepare output data for transmission and/or use by a user (e.g., as a response to an inference request). In at least one embodiment, inferencing tasks may be performed by one or more machine learning models, such as trained or deployed neural networks, which may include output models 4616 of training system 4604. [0253 - post-processing and a multi-format…] In at least one embodiment, for at least some media processing commands, command streamer 3303 supplies commands to a video front end 3334, which couples with a media engine 3337. In at least one embodiment, media engine 3337 includes a Video Quality Engine (VQE) 3330 for video and image post-processing and a multi-format encode/decode (MFX) 3333 engine to provide hardware-accelerated media data encode and decode. In at least one embodiment, geometry pipeline 3336 and media engine 3337 each generate execution threads for thread execution resources provided by at least one graphics core 3380A. [0295 - pre-processing, and/or post-processing…] In at least one embodiment fixed, function block 3930 also includes a graphics SoC interface 3937, a graphics microcontroller 3938, and a media pipeline 3939. In at least one embodiment fixed, graphics SoC interface 3937 provides an interface between graphics core 3900 and other processor cores within a system on a chip integrated circuit. In at least one embodiment, graphics microcontroller 3938 is a programmable sub-processor that is configurable to manage various functions of graphics processor 3900, including thread dispatch, scheduling, and pre-emption. In at least one embodiment, media pipeline 3939 includes logic to facilitate decoding, encoding, pre-processing, and/or post-processing of multimedia data, including image and video data. In at least one embodiment, media pipeline 3939 implements media operations via requests to compute or sampling logic within sub-cores 3901-3901F.[0395; 0398; 0401]); and wherein to execute the plurality of MLMs on one or more processing devices (Mallya et al. 2021/0142177 [0376 – execute machine learning models] In at least one embodiment, where a service 4620 includes an AI service (e.g., an inference service), one or more machine learning models associated with an application for anomaly detection (e.g., tumors, growth abnormalities, scarring, etc.) may be executed by calling upon (e.g., as an API call) an inference service (e.g., an inference server) to execute machine learning model(s), or processing thereof, as part of application execution. In at least one embodiment, where another application includes one or more machine learning models for segmentation tasks, an application may call upon an inference service to execute machine learning models for performing one or more of processing operations associated with segmentation tasks. In at least one embodiment, software 4618 implementing advanced processing and inferencing pipeline that includes segmentation application and anomaly detection application may be streamlined because each application may call upon a same inference service to perform one or more inferencing tasks.[0376; 0392; 0398 – execute machine learning models]), the processor is to cause the one or more processing devices to perform: the one or more pre-processing operations that precede the inference processing, or the one or more post-processing operations that are subsequent to the inference processing (Mallya et al. 2021/0142177 [0371 - pre-processing as part of data processing pipeline to prepare data for processing by one or more applications. … post-processing may be performed on an output of one or more inferencing tasks or other processing tasks…] In at least one embodiment, a data processing pipeline may receive input data (e.g., imaging data 4608) in a DICOM, RIS, CIS, REST compliant, RPC, raw, and/or other format in response to an inference request (e.g., a request from a user of deployment system 4606, such as a clinician, a doctor, a radiologist, etc.). In at least one embodiment, input data may be representative of one or more images, video, and/or other data representations generated by one or more imaging devices, sequencing devices, radiology devices, genomics devices, and/or other device types. In at least one embodiment, data may undergo pre-processing as part of data processing pipeline to prepare data for processing by one or more applications. In at least one embodiment, post-processing may be performed on an output of one or more inferencing tasks or other processing tasks of a pipeline to prepare an output data for a next application and/or to prepare output data for transmission and/or use by a user (e.g., as a response to an inference request). In at least one embodiment, inferencing tasks may be performed by one or more machine learning models, such as trained or deployed neural networks, which may include output models 4616 of training system 4604. [0253 - post-processing and a multi-format…] In at least one embodiment, for at least some media processing commands, command streamer 3303 supplies commands to a video front end 3334, which couples with a media engine 3337. In at least one embodiment, media engine 3337 includes a Video Quality Engine (VQE) 3330 for video and image post-processing and a multi-format encode/decode (MFX) 3333 engine to provide hardware-accelerated media data encode and decode. In at least one embodiment, geometry pipeline 3336 and media engine 3337 each generate execution threads for thread execution resources provided by at least one graphics core 3380A. [0295 - pre-processing, and/or post-processing…] In at least one embodiment fixed, function block 3930 also includes a graphics SoC interface 3937, a graphics microcontroller 3938, and a media pipeline 3939. In at least one embodiment fixed, graphics SoC interface 3937 provides an interface between graphics core 3900 and other processor cores within a system on a chip integrated circuit. In at least one embodiment, graphics microcontroller 3938 is a programmable sub-processor that is configurable to manage various functions of graphics processor 3900, including thread dispatch, scheduling, and pre-emption. In at least one embodiment, media pipeline 3939 includes logic to facilitate decoding, encoding, pre-processing, and/or post-processing of multimedia data, including image and video data. In at least one embodiment, media pipeline 3939 implements media operations via requests to compute or sampling logic within sub-cores 3901-3901F.[0395; 0398; 0401]).
18/229,929 – Claim 18. Mallya et al. 2021/0142177 further teaches The system of claim 11, wherein the one or more inference applications (Mallya et al. 2021/0142177 [0396 - inference applications] In at least one embodiment, transfer of requests between services 4620 and inference applications may be hidden behind a software development kit (SDK), and robust transport may be provide through a queue. In at least one embodiment, a request will be placed in a queue via an API for an individual application/tenant ID combination and an SDK will pull a request from a queue and give a request to an application. In at least one embodiment, a name of a queue may be provided in an environment from where an SDK will pick it up. In at least one embodiment, asynchronous communication through a queue may be useful as it may allow any instance of an application to pick up work as it becomes available. Results may be transferred back through a queue, to ensure no data is lost. In at least one embodiment, queues may also provide an ability to segment work, as highest priority work may go to a queue with most instances of an application connected to it, while lowest priority work may go to a queue with a single instance connected to it that processes tasks in an order received. In at least one embodiment, an application may run on a GPU-accelerated instance generated in cloud 4726, and an inference service may perform inferencing on a GPU.) comprise at least one inference backend capable of being selectively directed to execute an MLM (Mallya et al. 2021/0142177 [0376 – execute machine learning models] In at least one embodiment, where a service 4620 includes an AI service (e.g., an inference service), one or more machine learning models associated with an application for anomaly detection (e.g., tumors, growth abnormalities, scarring, etc.) may be executed by calling upon (e.g., as an API call) an inference service (e.g., an inference server) to execute machine learning model(s), or processing thereof, as part of application execution. In at least one embodiment, where another application includes one or more machine learning models for segmentation tasks, an application may call upon an inference service to execute machine learning models for performing one or more of processing operations associated with segmentation tasks. In at least one embodiment, software 4618 implementing advanced processing and inferencing pipeline that includes segmentation application and anomaly detection application may be streamlined because each application may call upon a same inference service to perform one or more inferencing tasks.[0376; 0392; 0398 – execute machine learning models]) using a graphics processing unit (GPU) and/or a central processing unit (CPU) (Mallya et al. 2021/0142177 [0080 – CPUs and GPUs] In at least one embodiment a processor 628 (or a processor of training manager 612 or inference module 618) will be a central processing unit (CPU). As mentioned, however, resources in such environments can utilize GPUs to process data for at least certain types of requests. With thousands of cores, GPUs are designed to handle substantial parallel workloads and, therefore, have become popular in deep learning for training neural networks and generating predictions. While use of GPUs for offline builds has enabled faster training of larger and more complex models, generating predictions offline implies that either request-time input features cannot be used or predictions must be generated for all permutations of features and stored in a lookup table to serve real-time requests. If a deep learning framework supports a CPU-mode and a model is small and simple enough to perform a feed-forward on a CPU with a reasonable latency, then a service on a CPU instance could host a model. In this case, training can be done offline on a GPU and inference done in real-time on a CPU. If a CPU approach is not viable, then a service can run on a GPU instance. Because GPUs have different performance and cost characteristics than CPUs, however, running a service that offloads a runtime algorithm to a GPU can require it to be designed differently from a CPU based service.[0114-0115]), and wherein the inference backend is further capable of being selectively directed to execute multiple MLMs sequentially (Mallya et al. 2021/0142177 [0385 - … include any number of applications that may be sequentially, non-sequentially, or otherwise applied to…] In at least one embodiment, deployment system 4606 may execute deployment pipelines 4710. In at least one embodiment, deployment pipelines 4710 may include any number of applications that may be sequentially, non-sequentially, or otherwise applied to imaging data (and/or other data types) generated by imaging devices, sequencing devices, genomics devices, etc.—including AI-assisted annotation, as described above. In at least one embodiment, as described herein, a deployment pipeline 4710 for an individual device may be referred to as a virtual instrument for a device (e.g., a virtual ultrasound instrument, a virtual CT scan instrument, a virtual sequencing instrument, etc.). In at least one embodiment, for a single device, there may be more than one deployment pipeline 4710 depending on information desired from data generated by a device. In at least one embodiment, where detections of anomalies are desired from an MRI machine, there may be a first deployment pipeline 4710, and where image enhancement is desired from output of an MRI machine, there may be a second deployment pipeline 4710. [0080 – CPUs and GPUs] In at least one embodiment a processor 628 (or a processor of training manager 612 or inference module 618) will be a central processing unit (CPU). As mentioned, however, resources in such environments can utilize GPUs to process data for at least certain types of requests. With thousands of cores, GPUs are designed to handle substantial parallel workloads and, therefore, have become popular in deep learning for training neural networks and generating predictions. While use of GPUs for offline builds has enabled faster training of larger and more complex models, generating predictions offline implies that either request-time input features cannot be used or predictions must be generated for all permutations of features and stored in a lookup table to serve real-time requests. If a deep learning framework supports a CPU-mode and a model is small and simple enough to perform a feed-forward on a CPU with a reasonable latency, then a service on a CPU instance could host a model. In this case, training can be done offline on a GPU and inference done in real-time on a CPU. If a CPU approach is not viable, then a service can run on a GPU instance. Because GPUs have different performance and cost characteristics than CPUs, however, running a service that offloads a runtime algorithm to a GPU can require it to be designed differently from a CPU based service.[0114-0115]) or in parallel (Mallya et al. 2021/0142177 [0202 - FIGS. 25-26 illustrate additional exemplary graphics processor logic … FIG. 26 illustrates a highly-parallel general-purpose graphics processing unit…] FIGS. 25-26 illustrate additional exemplary graphics processor logic according to embodiments described herein. FIG. 25 illustrates a graphics core 2500 that may be included within graphics processor 2210 of FIG. 22, in at least one embodiment, and may be a unified shader core 2355A-2355N as in FIG. 24 in at least one embodiment. FIG. 26 illustrates a highly-parallel general-purpose graphics processing unit 2530 suitable for deployment on a multi-chip module in at least one embodiment. [0080 – CPUs and GPUs] In at least one embodiment a processor 628 (or a processor of training manager 612 or inference module 618) will be a central processing unit (CPU). As mentioned, however, resources in such environments can utilize GPUs to process data for at least certain types of requests. With thousands of cores, GPUs are designed to handle substantial parallel workloads and, therefore, have become popular in deep learning for training neural networks and generating predictions. While use of GPUs for offline builds has enabled faster training of larger and more complex models, generating predictions offline implies that either request-time input features cannot be used or predictions must be generated for all permutations of features and stored in a lookup table to serve real-time requests. If a deep learning framework supports a CPU-mode and a model is small and simple enough to perform a feed-forward on a CPU with a reasonable latency, then a service on a CPU instance could host a model. In this case, training can be done offline on a GPU and inference done in real-time on a CPU. If a CPU approach is not viable, then a service can run on a GPU instance. Because GPUs have different performance and cost characteristics than CPUs, however, running a service that offloads a runtime algorithm to a GPU can require it to be designed differently from a CPU based service.[0114-0115]).
18/229,929 – Claim 19. Mallya et al. 2021/0142177 further teaches The system of claim 11, wherein the system is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system for generating or presenting at least one of augmented reality content, virtual reality content, or mixed reality content; a system implemented using a robot; a system for performing conversational AI operations; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs) (Mallya et al. 2021/0142177 Virtual Machines (VMs) [0023; 0038; 0060; 0063]); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.
Claim 19 is further rejected under 35 U.S.C. 103 as being unpatentable over: Mallya et al. 2021/0142177; in view of Kuo et al. 2019/0156246; in view of Puri et al. 2022/0101047.
18/229,929 – Claim 19. Mallya et al. 2021/0142177 further teaches The system of claim 11, wherein the system is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system for generating or presenting at least one of augmented reality content, virtual reality content, or mixed reality content; a system implemented using a robot; a system for performing conversational AI operations; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs) (Mallya et al. 2021/0142177 Virtual Machines (VMs) [0023; 0038; 0060; 0063]); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.
----------------------------Claim Interpretation------------------------
The Office interprets all the elements of Claim 19 to be ‘OR’ elements. Thus, if any element is taught by the Primary Reference Mallya et al. 2021/0142177, then the claim is rejected. As Mallya et al. 2021/0142177 teaches at least one of the many elements (e.g., the VM element), then, the claim is rejected. Secondary Reference Puri et al. 2022/0101047 is only presented to show another reference that teaches many of the claimed alternative elements. It is noted that both references are assigned to NVIDIA Corporation.
Mallya et al. 2021/0142177 may not expressly disclose all of the alternative features (e.g., deep learning, robot, autonomous or semi-autonomous machine, etc…), however, Puri et al. 2022/0101047 teaches (Puri et al. 2022/0101047 [0024 - performing conversational AI or personal assistance operations, a system for performing simulation operations, a system for performing simulation operations to test or validate autonomous machine applications, a system for performing deep learning operations, a system implemented using an edge device, a system incorporating one or more Virtual Machines (VMs), a system implemented at least partially in a data center, or a system implemented at least partially using cloud computing resources] As various examples, disclosed techniques may be implemented in systems that comprise or are included in one or more of a system for performing conversational AI or personal assistance operations, a system for performing simulation operations, a system for performing simulation operations to test or validate autonomous machine applications, a system for performing deep learning operations, a system implemented using an edge device, a system incorporating one or more Virtual Machines (VMs), a system implemented at least partially in a data center, or a system implemented at least partially using cloud computing resources. [0115 - a virtual machine, a drone, a robot, a handheld communications device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, an edge device, any combination of these delineated devices, or any other suitable device] The client device(s) may include at least some of the components, features, and functionality of the example computing device(s) 900 described herein with respect to FIG. 9. By way of example and not limitation, a client device may be embodied as a Personal Computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smart watch, a wearable computer, a Personal Digital Assistant (PDA), an MP3 player, a virtual reality headset, a Global Positioning System (GPS) or device, a video player, a video camera, a surveillance device or system, a vehicle, a boat, a flying vessel, a virtual machine, a drone, a robot, a handheld communications device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, an edge device, any combination of these delineated devices, or any other suitable device. [0149 - deep learning applications] The accelerator(s) 1114 (e.g., the hardware acceleration cluster) may include a deep learning accelerator(s) (DLA). The DLA(s) may include one or more Tensor processing units (TPUs) that may be configured to provide an additional ten trillion operations per second for deep learning applications and inferencing. The TPUs may be accelerators configured to, and optimized for, performing image processing functions (e.g., for CNNs, RCNNs, etc.). The DLA(s) may further be optimized for a specific set of neural network types and floating point operations, as well as inferencing. The design of the DLA(s) may provide more performance per millimeter than a general-purpose GPU, and vastly exceeds the performance of a CPU. The TPU(s) may perform several functions, including a single-instance convolution function, supporting, for example, INT8, INT16, and FP16 data types for both features and weights, as well as post-processor functions. [Claim 22 - a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system incorporating one or more Virtual Machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources] 22. The system of claim 18, wherein the operations are performed by at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system incorporating one or more Virtual Machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Mallya et al. 2021/0142177 to include the features as taught by Puri et al. 2022/0101047. One of ordinary skill in the art would have been motivated to do so to implement well known tools and features useful for implementing multi-model inferencing frameworks and application programming interfaces which should prove to improve user experience, maximize profits, and optimize revenue.
Examiner’s Response to Arguments
Per Applicants’ amendments/arguments, the rejections are withdrawn.
Applicant's arguments have been considered but are moot in view of the new ground(s) of rejection.
Applicants’ amendments have necessitated the new grounds of rejection noted above.
Examiner’s Response: Claim Rejections – 35 USC §112
Per Applicants’ amendments/arguments, the rejections are withdrawn.
Applicant's arguments have been considered but are moot in view of the new ground(s) of rejection.
Applicants’ amendments have necessitated the new grounds of rejection noted above.
Examiner’s Response: Claim Rejections – 35 USC §101
Per Applicants’ amendments/arguments, the rejections are withdrawn. See notes above for additional reasoning and rationale for dropping 35 USC 101 rejection including Applicant’s amendments, arguments, lack of abstract idea, and practical integration.
Applicant's arguments have been considered but are moot in view of the new ground(s) of rejection.
Applicants’ amendments have necessitated the new grounds of rejection noted above.
Regarding Claims 1-20, on page(s) 8-11 of Applicant’s Remarks (dated 06/04/2026), Applicants traverse the 35 USC §101 rejections. Respectfully, the Office maintains that the invention is abstract and does not amount to significantly more and is therefore rejected as patent ineligible under 35 USC §101.
The claims are abstract
Applicant’s arguments have been fully considered but are not persuasive. The assertion that the claims do not recite a “method of organizing human activity” does not overcome the § 101 rejection. As explained in MPEP § 2106.04(a), the enumerated examples of organizing human activity are not exhaustive, and the abstract-idea analysis is not limited to those specific sub-categories. Claims may still be directed to an abstract idea even if they do not explicitly recite hedging, insurance, contracts, or similar activities.
The focus of claim 1 is on receiving user-provided configuration parameters, configuring a machine-learning inference pipeline based on those parameters, executing the pipeline, and returning results. These steps constitute data processing and information manipulation, which the courts have consistently held to be abstract ideas. See Electric Power Group, Content Extraction, BSG Tech, and SAP v. InvestPic. The claim does not recite any improvement to computer functionality, any unconventional computing component, or any technological advance in machine-learning models or pipeline execution. Instead, the claim relies on generic computing elements performing routine operations such as receiving data, configuring workflows, and executing known inference techniques.
Electric Power Group, LLC v. Alstom S.A. (2016)
The Federal Circuit affirmed the district court’s dismissal of claims to real-time power grid monitoring systems. The claims described collecting and analyzing data from multiple sources to detect and analyze grid events. The court found the claims patent ineligible because they were so result-focused and functional that they effectively covered any solution to an identified problem — a hallmark of an abstract idea.
Key takeaway: Even if a claim uses technical data and analysis, if it is merely a generic method of applying known data to a problem, it may fail the “significantly more” step of Alice.
Content Extraction and Transmission LLC v. Shearson Lehman Hutton Inc. (2003)
While not a Federal Circuit case, Content Extraction is often cited in the context of early § 101 jurisprudence. It involved a claim to a method for extracting and transmitting content from a database. The court held the claim ineligible because it was a generic method of data transmission without sufficient inventive concept.
Relevance: Reinforces that merely automating a known process or using a computer to perform a known function does not make it patent-eligible.
BSG Tech, LLC v. Bilski (2011)
In Bilski, the Federal Circuit (with the Supreme Court) held that a method for hedging commodity risk was ineligible because it was a generic method of doing business. The court emphasized that claims to methods of doing business are not patentable unless they include an inventive concept that transforms the method into a patent-eligible application.
Relevance: Supports the idea that abstract ideas, including business methods, require more than a generic application to be eligible.
SAP America, Inc. v. InvestPic, LLC (2018)
The Federal Circuit held that SAP’s claims to a method for generating and displaying a “business intelligence” dashboard were patent ineligible because they were directed to an abstract idea — a generic method of displaying data — and the additional elements were well-understood, routine, and conventional. The court noted that the claims did not integrate the idea into a practical application.
Key takeaway: Even if a claim uses a computer, if the computer is merely a tool to perform a known function, it does not overcome the abstract idea hurdle.
Applicant’s characterization of the claims as “specific technical operations” is not supported by the claim language. The claims do not recite a new API structure, a new pipeline architecture, a new data format, or any technical solution that improves the functioning of a computer or machine-learning system. Rather, the claims merely automate a conceptual process of selecting, arranging, and executing inference components—an abstract idea under established precedent.
Because the claims are directed to abstract data processing and do not include additional elements amounting to significantly more than the abstract idea itself, the rejection under 35 U.S.C. § 101 is maintained.
Furthermore, Applicant’s own specification supports this conclusion. For example, the specification indicates that the Machine learning has become a staple in a variety of industries and activities such as in medical imaging [0015] machine control, robotics, security, autonomous driving [0018] and systems for content creation [0019]. All of these and more listed in the specification are considered abstract ideas under certain methods of organizing human activities – commercial interactions including marketing or sales activities or behaviors and business relations.
The claims are NOT integrated into a practical application
Applicant argues that the claims are integrated into a practical application because they allegedly recite a “specific technical solution” that improves computer functionality, analogizing the claims to Enfish and McRO. This argument is not persuasive.
The Claims Do Not Improve the Functioning of a Computer or Any Other Technology
The recited steps—receiving configuration parameters via an API, identifying storage locations and inference applications, executing multiple machine-learning models, and rendering combined output—describe generic data processing performed by conventional computer components. The claims do not recite any improvement to how the computer stores, retrieves, processes, or manages data. They do not modify the operation of the inference applications, the processing devices, or the machine-learning models. Instead, they merely use known computer elements as tools to perform an abstract orchestration of model execution.
As such, the claims do not improve the functioning of the computer itself, nor do they improve another technology or technical field, as required by MPEP §2106.05(a).
Enfish Is Not Applicable
Applicant’s reliance on Enfish is misplaced. In Enfish, the claims were directed to a specific, self-referential table that improved the computer’s ability to store and retrieve data. The Federal Circuit emphasized that the invention changed the way the computer operated at a fundamental level.
Here, the claims do not recite any new data structure, any improvement to memory architecture, or any enhancement to computer operation. They merely invoke conventional components to perform routine machine-learning tasks. Accordingly, Enfish does not support eligibility.
McRO Is Not Applicable
Applicant’s analogy to McRO is also unpersuasive. In McRO, the claims recited specific rules that automated a previously manual animation process, resulting in a technological improvement not achievable by generic computers using conventional techniques.
The present claims do not recite any specific rules, algorithms, or technological improvements to model execution. They do not improve the models themselves, the inference applications, or the underlying computing hardware. Instead, they describe high-level functional steps—selecting models, executing them, and combining outputs—without specifying how those steps are performed in a manner that improves technology. The Federal Circuit has consistently held that such result-oriented functional claiming does not constitute a technological improvement.
The Claimed “Framework” Is Merely an Abstract Idea Implemented on Generic Computer Components
The claims describe an abstract workflow for orchestrating multiple machine-learning models. The recited operations—receiving parameters, identifying resources, executing models, and rendering output—are all conventional and can be performed using generic computing resources. The claims do not impose any meaningful limitations that tie the abstract idea to a practical application.
No Integration Into a Practical Application Under MPEP §2106.05
The claims fail to demonstrate:
an improvement to computer functionality,
an improvement to another technology or technical field,
a transformation of data into a different state or thing, or
any meaningful limitation beyond the abstract idea itself.
Thus, the claims do not integrate the abstract idea into a practical application.
Conclusion
Because the claims merely use generic computer components to perform abstract model-selection and execution steps, without improving the functioning of the computer or any other technology, they do not satisfy the “practical application” requirement. Applicant’s reliance on Enfish and McRO is misplaced, as those cases involved specific technological improvements not present in the instant claims. Accordingly, the claims remain directed to an abstract idea and do not demonstrate integration into a practical application.
Under Step 2B, there is NO ordered combination that provides significantly more.
Applicant contends that the ordered combination of claim elements provides “significantly more” than any alleged abstract idea because the claims recite a “specific pipeline architecture” for multi-model inference. This argument is not persuasive.
The Ordered Combination Merely Arranges Conventional Computer Functions
The recited steps—receiving configuration parameters via an API, identifying storage locations and inference applications, configuring those applications, executing multiple machine-learning models, and rendering combined output—are all well-understood, routine, and conventional operations performed by generic computer components. Each step reflects standard machine-learning workflow orchestration, and the combination of these steps does not amount to an unconventional arrangement of elements.
The claims do not recite any improvement to the functioning of the computer, any specialized hardware, any new model-execution technique, or any non-conventional data processing architecture. Instead, they simply arrange known operations in a predictable sequence to achieve a result—executing multiple models and combining their outputs—that is itself conventional in the field.
Functional Claiming Without Technological Detail Does Not Provide “Significantly More”
Applicant asserts that the claims do not merely “apply it on a computer,” but the claims recite only result-oriented functional language describing what the system is intended to accomplish, not how it achieves those results in a non-conventional manner. The Federal Circuit has repeatedly held that such functional recitations do not transform an abstract idea into patent-eligible subject matter.
The claims do not specify:
any particular improvement to model execution,
any unconventional configuration of inference applications,
any new data structure or processing technique, or
any technological advance in multi-model inference.
Absent such detail, the ordered combination remains a generic automation of an abstract idea.
The Alleged “Pipeline Architecture” Is Merely an Abstract Workflow
Applicant characterizes the claims as reciting a “specific pipeline architecture,” but the claims do not define any particular architecture beyond a sequence of conventional steps. The mere presence of a pipeline or workflow does not confer eligibility when the pipeline consists of routine operations performed by generic computing components.
Courts have consistently held that organizing conventional steps into a workflow—even a multi-step workflow—does not provide an inventive concept when the steps themselves are routine and the combination is predictable.
No Inventive Concept Is Present in the Claimed Combination
Under Step 2B, the question is whether the additional elements, individually or in combination, amount to “significantly more” than the abstract idea. Here, they do not.
The claims fail to recite:
an improvement to computer functionality,
an unconventional technical solution,
a non-routine combination of elements, or
any limitation that meaningfully restricts the abstract idea.
Instead, the claims simply use generic computer components to implement an abstract concept—coordinating multiple machine-learning models and combining their outputs.
Conclusion Under Step 2B
Because the ordered combination of elements reflects only well-understood, routine, and conventional computer functions arranged in a predictable manner, it does not supply an inventive concept. Applicant’s characterization of the claims as a “specific technical approach” is not supported by the claim language, which recites only high-level functional steps typical of conventional machine-learning pipelines.
Accordingly, the claims do not provide “significantly more” under Step 2B and remain ineligible.
Preemption
Applicant argues that the Examiner’s discussion of preemption is improper because preemption is “not a standalone test” and because the claims allegedly recite a specific, limited approach to multi-model inferencing. Applicant’s argument is not persuasive.
The Examiner’s Preemption Analysis Is Proper Under the Alice/Mayo Framework
Applicant correctly notes that preemption is not a standalone test. However, the MPEP also makes clear that preemption concerns remain relevant when determining whether a claim is directed to an abstract idea and whether additional elements amount to significantly more. See MPEP § 2106.04. The Examiner’s analysis was made within the context of the Alice/Mayo framework, not as an independent test. As such, the Examiner’s discussion of preemption is entirely consistent with MPEP guidance and Federal Circuit precedent.
The Claims Are Drafted at a High Level of Abstraction and Risk Monopolizing the Abstract Idea
Applicant asserts that the claims are limited to a “specific approach” involving a user API, configuration parameters, inference applications, and rendering combined output. However, the claims recite these elements only in generic, functional terms without specifying any particular technological implementation or unconventional technique.
The claims do not recite:
any specific improvement to model execution,
any particular data structure or processing mechanism,
any specialized hardware,
or any non-conventional configuration of inference applications.
Instead, the claims broadly cover the concept of orchestrating multiple machine-learning models and combining their outputs—an abstract idea—implemented using routine computer components. Because the claims are drafted at a high level of generality, they risk preempting all practical applications of the abstract idea of multi-model inference orchestration.
The Alleged “Specific Approach” Does Not Meaningfully Limit the Claims
Applicant argues that other approaches remain available, such as those that do not use a user API or that use different configuration mechanisms. This argument is not persuasive.
The Federal Circuit has repeatedly held that reciting generic computer components or conventional variations of an abstract workflow does not meaningfully limit a claim or avoid preemption. See, e.g., Alice, Mayo, OIP Tech., Electric Power Group, ChargePoint. Merely specifying that the abstract idea is carried out using a user API, configuration parameters, and inference applications does not impose a meaningful restriction on the scope of the claims.
The claims still cover the abstract concept of:
receiving model-selection information,
executing multiple models, and
combining their outputs,
Regardless of the specific implementation details, such functional claiming does not avoid preemption.
Preemption Concerns Support the Examiner’s Eligibility Determination
Under the Alice/Mayo framework, preemption concerns help confirm that the claims are directed to an abstract idea and lack an inventive concept. Because the claims recite only generic computer components performing routine machine-learning operations, they do not meaningfully limit the abstract idea. As a result, the claims would effectively preempt the field of multi-model inference orchestration implemented on conventional computing systems.
Thus, the Examiner’s preemption analysis is appropriate and supports the conclusion that the claims remain ineligible.
Conclusion
Applicant’s argument that the Examiner’s preemption discussion is improper is not persuasive. The Examiner’s analysis is consistent with MPEP § 2106.04 and Federal Circuit precedent, and the claims—drafted in broad, functional terms—risk monopolizing the abstract idea of orchestrating multiple machine-learning models and combining their outputs. The alleged “specific approach” does not meaningfully limit the claims or avoid preemption. Accordingly, the claims remain directed to an abstract idea and do not recite significantly more.
Examiner’s Response: Claim Rejections – 35 USC § 103
Per Applicants’ amendments/arguments, the rejections are withdrawn. See notes above for additional reasoning and rationale for dropping prior-art rejection including Applicant’s amendments and arguments and unique combination of features and elements not taught by the prior-art without hindsight reasoning.
Applicant's arguments have been considered but are moot in view of the new ground(s) of rejection.
Applicants’ amendments have necessitated the new grounds of rejection noted above.
Regarding Claim 1, on pages 12-16 of Applicant’s Remarks (dated 06/04/2026), Applicant(s) argues that the cited references fails to teach, describe, or suggest the amended features.
Specifically, Applicant argues the following:
Fails to connect the mapped "configuration parameters" to the "storage locations of the plurality of MLMs";
conflates training data origins with inference input data locations; and
mapping of "combined representation" misapplies the reference.
With respect, Applicant’s arguments are deemed unpersuasive, and the amended feature(s) remain rejected as follows.
Under the Broadest Reasonable Interpretation (BRI) of the cited references the Office maintains the rejections. Furthermore, Applicant is arguing what they have not claimed. The Office maintains the rejection as previously noted above and furthermore herein below.
18/229,929 – Claim 1. Mallya et al. 2021/0142177 teaches A method comprising: receiving, via a user application programming interface (API) (Mallya et al. 2021/0142177 [0079 - requests may be received through a user interface] In at least one embodiment, at a subsequent point in time, a request may be received from client device 602 (or another such device) for content (e.g., path determinations) or data that is at least partially determined or impacted by a trained neural network. This request can include, for example, input data to be processed using a neural network to obtain one or more inferences or other output values, classifications, or predictions. In at least one embodiment, input data can be received to interface layer 608 and directed to inference module 618, although a different system or service can be used as well. In at least one embodiment, inference module 618 can obtain an appropriate trained network, such as a trained deep neural network (DNN) as discussed herein, from model repository 616 if not already stored locally to inference module 618. Inference module 618 can provide data as input to a trained network, which can then generate one or more inferences as output. This may include, for example, a classification of an instance of input data. In at least one embodiment, inferences can then be transmitted to client device 602 for display or other communication to a user. In at least one embodiment, context data for a user may also be stored to a user context data repository 622, which may include data about a user which may be useful as input to a network in generating inferences, or determining data to return to a user after obtaining instances. In at least one embodiment, relevant data, which may include at least some of input or inference data, may also be stored to a local database 620 for processing future requests. In at least one embodiment, a user can use account or other information to access resources or functionality of a provider environment. In at least one embodiment, if permitted and available, user data may also be collected and used to further train models, in order to provide more accurate inferences for future requests. In at least one embodiment, requests may be received through a user interface to a machine learning application 626 executing on client device 602, and results displayed through a same interface. A client device can include resources such as a processor 628 and memory 630 for generating a request and processing results or a response, as well as at least one data storage element 632 for storing data for machine learning application 626. [0128 - graphical user interfaces] FIG. 12A is a block diagram illustrating an exemplary computer system, which may be a system with interconnected devices and components, a system-on-a-chip (SOC) or some combination thereof 1200 formed with a processor that may include execution units to execute an instruction, according to at least one embodiment. In at least one embodiment, computer system 1200 may include, without limitation, a component, such as a processor 1202 to employ execution units including logic to perform algorithms for process data, in accordance with present disclosure, such as in embodiment described herein. In at least one embodiment, computer system 1200 may include processors, such as PENTIUM® Processor family, Xeon™, Itanium®, XScale™ and/or StrongARM™, Intel® Core™, or Intel® Nervana™ microprocessors available from Intel Corporation of Santa Clara, Calif., although other systems (including PCs having other microprocessors, engineering workstations, set-top boxes and like) may also be used. In at least one embodiment, computer system 1200 may execute a version of WINDOWS' operating system available from Microsoft Corporation of Redmond, Wash., although other operating systems (UNIX and Linux for example), embedded software, and/or graphical user interfaces, may also be used. [0135 - user input and keyboard interfaces] In at least one embodiment, computer system 1200 may use system I/O 1222 that is a proprietary hub interface bus to couple MCH 1216 to I/O controller hub (“ICH”) 1230. In at least one embodiment, ICH 1230 may provide direct connections to some I/O devices via a local I/O bus. In at least one embodiment, local I/O bus may include, without limitation, a high-speed I/O bus for connecting peripherals to memory 1220, chipset, and processor 1202. Examples may include, without limitation, an audio controller 1229, a firmware hub (“flash BIOS”) 1228, a wireless transceiver 1226, a data storage 1224, a legacy I/O controller 1223 containing user input and keyboard interfaces 1225, a serial expansion port 1227, such as Universal Serial Bus (“USB”), and a network controller 1234. data storage 1224 may comprise a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or other mass storage device. [0393 - a request may be received by a set of API…] In at least one embodiment, shared storage may be mounted to AI services 4718 within system 4700. In at least one embodiment, shared storage may operate as a cache (or other storage device type) and may be used to process inference requests from applications. In at least one embodiment, when an inference request is submitted, a request may be received by a set of API instances of deployment system 4606, and one or more instances may be selected (e.g., for best fit, for load balancing, etc.) to process a request. In at least one embodiment, to process a request, a request may be entered into a database, a machine learning model may be located from model registry 4624 if not already in a cache, a validation step may ensure appropriate machine learning model is loaded into a cache (e.g., shared storage), and/or a copy of a model may be saved to a cache. In at least one embodiment, a scheduler (e.g., of pipeline manager 4712) may be used to launch an application that is referenced in a request if an application is not already running or if there are not enough instances of an application. In at least one embodiment, if an inference server is not already launched to execute a model, an inference server may be launched. Any number of inference servers may be launched per model. In at least one embodiment, in a pull model, in which inference servers are clustered, models may be cached whenever load balancing is advantageous. In at least one embodiment, inference servers may be statically loaded in corresponding, distributed servers.), configuration parameters (Mallya et al. 2021/0142177 [0121 - as shown in FIG. 11, framework layer 1120 includes a job scheduler 1122, a configuration manager 1124, a resource manager… configuration manager 1124 may be capable of configuring different layers such as software layer 1130 and framework layer… (interpreted as configuration parameters)][0126 - software and/or hardware resources described above may be configured as a service to allow users to train or performing inferencing of information… (interpreted as configuration parameters)][0100 - hyperparameter configurations] In at least one embodiment, instances of a dataset can be embedded into a lower dimensional space of a certain size during pre-processing. In at least one embodiment, a size of this space is a parameter to be tuned. In at least one embodiment, an architecture of a CNN contains many tunable parameters. A parameter for filter sizes can represent an interpretation of information that corresponds to a size of an instance that will be analyzed. In computational linguistics, this is known as an n-gram size. An example CNN uses three different filter sizes, which represent potentially different n-gram sizes. A number of filters per filter size can correspond to a depth of a filter. Each filter attempts to learn something different from a structure of an instance, such as a sentence structure for textual data. In a convolutional layer, an activation function can be a rectified linear unit and a pooling type set as max pooling. Results can then be concatenated into a single dimensional vector, and a last layer is fully connected onto a two-dimensional output. This corresponds to a binary classification to which an optimization function can be applied. One such function is an implementation of a Root Mean Square (RMS) propagation method of gradient descent, where example hyperparameters can include learning rate, batch size, maximum gradient normal, and epochs. With neural networks, regularization can be an extremely important consideration. In at least one embodiment input data may be relatively sparse. A main hyperparameter in such a situation can be a dropout at a penultimate layer, which represents a proportion of nodes that will not “fire” at each training cycle. An example training process can suggest different hyperparameter configurations based on feedback for a performance of previous configurations. This model can be trained with a proposed configuration, evaluated on a designated validation set, and performance reporting. This process can be repeated to, for example, trade off exploration (learning more about different configurations) and exploitation (leveraging previous knowledge to achieve better results). [0101 - configuration parameters] As training CNNs can be parallelized and GPU-enabled computing resources can be utilized, multiple optimization strategies can be attempted for different scenarios. A complex scenario allows tuning model architecture and preprocessing and stochastic gradient descent parameters. This expands a model configuration space. In a basic scenario, only preprocessing and stochastic gradient descent parameters are tuned. There can be a greater number of configuration parameters in a complex scenario than in a basic scenario. Tuning in a joint space can be performed using a linear or exponential number of steps, iteration through an optimization loop for models. A cost for such a tuning process can be significantly less than for tuning processes such as random search and grid search, without any significant performance loss.) for execution of a plurality of machine learning models (MLMs) (Mallya et al. 2021/0142177 [0334 - pipeline manager 4302 configures at least one of DPCs 4306 to implement a neural network model and/or a computing pipeline…][0376 – execute machine learning models] In at least one embodiment, where a service 4620 includes an AI service (e.g., an inference service), one or more machine learning models associated with an application for anomaly detection (e.g., tumors, growth abnormalities, scarring, etc.) may be executed by calling upon (e.g., as an API call) an inference service (e.g., an inference server) to execute machine learning model(s), or processing thereof, as part of application execution. In at least one embodiment, where another application includes one or more machine learning models for segmentation tasks, an application may call upon an inference service to execute machine learning models for performing one or more of processing operations associated with segmentation tasks. In at least one embodiment, software 4618 implementing advanced processing and inferencing pipeline that includes segmentation application and anomaly detection application may be streamlined because each application may call upon a same inference service to perform one or more inferencing tasks.[0383 - system 4700 may be configured to access and referenced data (e.g., DICOM data, RIS data, raw data, CIS data, REST compliant data, RPC data, raw data, etc.) from PACS servers (e.g., via a DICOM adapter 4702, or another data type adapter such as RIS, CIS, REST compliant, RPC, raw, etc.) to perform operations, such as training machine learning models, deploying machine learning models, image processing, inferencing, and/or other operations…][0376; 0392; 0398 – execute machine learning models]), wherein the configuration parameters identify (Mallya et al. 2021/0142177 [0101 - configuration parameters] As training CNNs can be parallelized and GPU-enabled computing resources can be utilized, multiple optimization strategies can be attempted for different scenarios. A complex scenario allows tuning model architecture and preprocessing and stochastic gradient descent parameters. This expands a model configuration space. In a basic scenario, only preprocessing and stochastic gradient descent parameters are tuned. There can be a greater number of configuration parameters in a complex scenario than in a basic scenario. Tuning in a joint space can be performed using a linear or exponential number of steps, iteration through an optimization loop for models. A cost for such a tuning process can be significantly less than for tuning processes such as random search and grid search, without any significant performance loss.): storage locations of the plurality of MLMs (Mallya et al. 2021/0142177 [0078 - a trained network can be stored to a model repository 616, for example, that may store different models or networks for users, applications, or services, etc. In at least one embodiment there may be multiple models for a single application or entity, as may be utilized based on a number of different factors…][0079 - data storage element 632 for storing data for machine learning application] In at least one embodiment, at a subsequent point in time, a request may be received from client device 602 (or another such device) for content (e.g., path determinations) or data that is at least partially determined or impacted by a trained neural network. This request can include, for example, input data to be processed using a neural network to obtain one or more inferences or other output values, classifications, or predictions. In at least one embodiment, input data can be received to interface layer 608 and directed to inference module 618, although a different system or service can be used as well. In at least one embodiment, inference module 618 can obtain an appropriate trained network, such as a trained deep neural network (DNN) as discussed herein, from model repository 616 if not already stored locally to inference module 618. Inference module 618 can provide data as input to a trained network, which can then generate one or more inferences as output. This may include, for example, a classification of an instance of input data. In at least one embodiment, inferences can then be transmitted to client device 602 for display or other communication to a user. In at least one embodiment, context data for a user may also be stored to a user context data repository 622, which may include data about a user which may be useful as input to a network in generating inferences, or determining data to return to a user after obtaining instances. In at least one embodiment, relevant data, which may include at least some of input or inference data, may also be stored to a local database 620 for processing future requests. In at least one embodiment, a user can use account or other information to access resources or functionality of a provider environment. In at least one embodiment, if permitted and available, user data may also be collected and used to further train models, in order to provide more accurate inferences for future requests. In at least one embodiment, requests may be received through a user interface to a machine learning application 626 executing on client device 602, and results displayed through a same interface. A client device can include resources such as a processor 628 and memory 630 for generating a request and processing results or a response, as well as at least one data storage element 632 for storing data for machine learning application 626. [0177 - memory location] In one embodiment, each WD 1984 is specific to a particular graphics acceleration module 1946 and/or graphics processing engines 1731-1732, N (shown in FIG. 17). It contains all information required by a graphics processing engine 1731-1732, N (shown in FIG. 17) to do work or it can be a pointer to a memory location where an application has set up a command queue of work to be completed.), storage locations of input data into the plurality of MLMs (Mallya et al. 2021/0142177 [0112; 0119][0368 - machine learning models may have been trained on imaging data from one location, two locations, or any number of locations] In at least one embodiment, training pipeline 4704 (FIG. 47) may include a scenario where facility 4602 needs a machine learning model for use in performing one or more processing tasks for one or more applications in deployment system 4606, but facility 4602 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, an existing machine learning model may be selected from a model registry 4624. In at least one embodiment, model registry 4624 may include machine learning models trained to perform a variety of different inference tasks on imaging data. In at least one embodiment, machine learning models in model registry 4624 may have been trained on imaging data from different facilities than facility 4602 (e.g., facilities remotely located). In at least one embodiment, machine learning models may have been trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, when being trained on imaging data from a specific location, training may take place at that location, or at least in a manner that protects confidentiality of imaging data or restricts imaging data from being transferred off-premises (e.g., to comply with HIPAA regulations, privacy regulations, etc.). In at least one embodiment, once a model is trained—or partially trained—at one location, a machine learning model may be added to model registry 4624. In at least one embodiment, a machine learning model may then be retrained, or updated, at any number of other facilities, and a retrained or updated model may be made available in model registry 4624. In at least one embodiment, a machine learning model may then be selected from model registry 4624—and referred to as output model 4616—and may be used in deployment system 4606 to perform one or more processing tasks for one or more applications of a deployment system. [0391 - data in same location of a memory may be used for any number of processing tasks] In at least one embodiment, services 4620 leveraged by and shared by applications or containers in deployment system 4606 may include compute services 4716, AI services 4718, visualization services 4720, and/or other service types. In at least one embodiment, applications may call (e.g., execute) one or more of services 4620 to perform processing operations for an application. In at least one embodiment, compute services 4716 may be leveraged by applications to perform super-computing or other high-performance computing (HPC) tasks. In at least one embodiment, compute service(s) 4716 may be leveraged to perform parallel processing (e.g., using a parallel computing platform 4730) for processing data through one or more of applications and/or one or more tasks of a single application, substantially simultaneously. In at least one embodiment, parallel computing platform 4730 (e.g., NVIDIA's CUDA) may enable general purpose computing on GPUs (GPGPU) (e.g., GPUs 4722). In at least one embodiment, a software layer of parallel computing platform 4730 may provide access to virtual instruction sets and parallel computational elements of GPUs, for execution of compute kernels. In at least one embodiment, parallel computing platform 4730 may include memory and, in some embodiments, a memory may be shared between and among multiple containers, and/or between and among different processing tasks within a single container. In at least one embodiment, inter-process communication (IPC) calls may be generated for multiple containers and/or for multiple processes within a container to use same data from a shared segment of memory of parallel computing platform 4730 (e.g., where multiple different stages of an application or multiple applications are processing same information). In at least one embodiment, rather than making a copy of data and moving data to different locations in memory (e.g., a read/write operation), same data in same location of a memory may be used for any number of processing tasks (e.g., at a same time, at different times, etc.). In at least one embodiment, as data is used to generate new data as a result of processing, this information of a new location of data may be stored and shared between various applications. In at least one embodiment, location of data and a location of updated or modified data may be part of a definition of how a payload is understood within containers.), and one or more inference applications (Mallya et al. 2021/0142177 [0396 - inference applications] In at least one embodiment, transfer of requests between services 4620 and inference applications may be hidden behind a software development kit (SDK), and robust transport may be provide through a queue. In at least one embodiment, a request will be placed in a queue via an API for an individual application/tenant ID combination and an SDK will pull a request from a queue and give a request to an application. In at least one embodiment, a name of a queue may be provided in an environment from where an SDK will pick it up. In at least one embodiment, asynchronous communication through a queue may be useful as it may allow any instance of an application to pick up work as it becomes available. Results may be transferred back through a queue, to ensure no data is lost. In at least one embodiment, queues may also provide an ability to segment work, as highest priority work may go to a queue with most instances of an application connected to it, while lowest priority work may go to a queue with a single instance connected to it that processes tasks in an order received. In at least one embodiment, an application may run on a GPU-accelerated instance generated in cloud 4726, and an inference service may perform inferencing on a GPU.) for performing inference processing (Mallya et al. 2021/0142177 [0073; 0079][0114 - inference processing] In at least one embodiment, activation storage 1020 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., Flash memory), or other storage. In at least one embodiment, activation storage 1020 may be completely or partially within or external to one or more processors or other logical circuits. In at least one embodiment, choice of whether activation storage 1020 is internal or external to a processor, for example, or comprised of DRAM, SRAM, Flash or some other storage type may depend on available storage on-chip versus off-chip, latency requirements of training and/or inferencing functions being performed, batch size of data used in inferencing and/or training of a neural network, or some combination of these factors. In at least one embodiment, inference and/or training logic 1015 illustrated in FIG. 9 may be used in conjunction with an application-specific integrated circuit (“ASIC”), such as Tensorflow® Processing Unit from Google, an inference processing unit (IPU) from Graphcore™, or a Nervana® (e.g., “Lake Crest”) processor from Intel Corp. In at least one embodiment, inference and/or training logic 1015 illustrated in FIG. 9 may be used in conjunction with central processing unit (“CPU”) hardware, graphics processing unit (“GPU”) hardware or other hardware, such as field programmable gate arrays (“FPGAs”). [0115 - inference processing] FIG. 10 illustrates inference and/or training logic 1015, according to at least one or more embodiments. In at least one embodiment, inference and/or training logic 1015 may include, without limitation, hardware logic in which computational resources are dedicated or otherwise exclusively used in conjunction with weight values or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, inference and/or training logic 1015 illustrated in FIG. 10 may be used in conjunction with an application-specific integrated circuit (ASIC), such as Tensorflow® Processing Unit from Google, an inference processing unit (IPU) from Graphcore™, or a Nervana® (e.g., “Lake Crest”) processor from Intel Corp. In at least one embodiment, inference and/or training logic 1015 illustrated in FIG. 10 may be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware or other hardware, such as field programmable gate arrays (FPGAs). In at least one embodiment, inference and/or training logic 1015 includes, without limitation, code and/or data storage 1001 and code and/or data storage 1005, which may be used to store code (e.g., graph code), weight values and/or other information, including bias values, gradient information, momentum values, and/or other parameter or hyperparameter information. In at least one embodiment illustrated in FIG. 10, each of code and/or data storage 1001 and code and/or data storage 1005 is associated with a dedicated computational resource, such as computational hardware 1002 and computational hardware 1006, respectively. In at least one embodiment, each of computational hardware 1002 and computational hardware 1006 comprises one or more ALUs that perform mathematical functions, such as linear algebraic functions, only on information stored in code and/or data storage 1001 and code and/or data storage 1005, respectively, result of which is stored in activation storage 1020.) of the input data using the plurality of MLMs (Mallya et al. 2021/0142177 [0079 – input data; 0085 – input data][0082; 0084; 0381][Fig. 9; 0012 - inference and/or training logic] FIG. 9 illustrates inference and/or training logic, according to at least one embodiment; [0013 - FIG. 10 illustrates inference and/or training logic] FIG. 10 illustrates inference and/or training logic, according to at least one embodiment; [0073 - Once a DNN is trained, this DNN can be deployed and used to identify and classify objects or patterns in a process known as inference. Examples of inference (a process through which a DNN extracts useful information from a given input)] A deep neural network (DNN) model includes multiple layers of many connected perceptrons (e.g., nodes) that can be trained with enormous amounts of input data to quickly solve complex problems with high accuracy. In one example, a first layer of a DNN model breaks down an input image of an automobile into various sections and looks for basic patterns such as lines and angles. Second layer assembles lines to look for higher-level patterns such as wheels, windshields, and mirrors. A next layer identifies a type of vehicle, and a final few layers generate a label for an input image, identifying a model of a specific automobile brand. Once a DNN is trained, this DNN can be deployed and used to identify and classify objects or patterns in a process known as inference. Examples of inference (a process through which a DNN extracts useful information from a given input) include identifying handwritten numbers on checks deposited into KIM machines, identifying images of friends in photos, delivering movie recommendations, identifying and classifying different types of automobiles, pedestrians, and road hazards in driverless cars, or translating human speech in near real-time. [0079 - input data to be processed using a neural network to obtain one or more inferences or other output values, classifications, or predictions. In at least one embodiment, input data can be received to interface layer 608 and directed to inference module] In at least one embodiment, at a subsequent point in time, a request may be received from client device 602 (or another such device) for content (e.g., path determinations) or data that is at least partially determined or impacted by a trained neural network. This request can include, for example, input data to be processed using a neural network to obtain one or more inferences or other output values, classifications, or predictions. In at least one embodiment, input data can be received to interface layer 608 and directed to inference module 618, although a different system or service can be used as well. In at least one embodiment, inference module 618 can obtain an appropriate trained network, such as a trained deep neural network (DNN) as discussed herein, from model repository 616 if not already stored locally to inference module 618. Inference module 618 can provide data as input to a trained network, which can then generate one or more inferences as output. This may include, for example, a classification of an instance of input data. In at least one embodiment, inferences can then be transmitted to client device 602 for display or other communication to a user. In at least one embodiment, context data for a user may also be stored to a user context data repository 622, which may include data about a user which may be useful as input to a network in generating inferences, or determining data to return to a user after obtaining instances. In at least one embodiment, relevant data, which may include at least some of input or inference data, may also be stored to a local database 620 for processing future requests. In at least one embodiment, a user can use account or other information to access resources or functionality of a provider environment. In at least one embodiment, if permitted and available, user data may also be collected and used to further train models, in order to provide more accurate inferences for future requests. In at least one embodiment, requests may be received through a user interface to a machine learning application 626 executing on client device 602, and results displayed through a same interface. A client device can include resources such as a processor 628 and memory 630 for generating a request and processing results or a response, as well as at least one data storage element 632 for storing data for machine learning application 626. [0107 - FIG. 9 illustrates inference and/or training logic 915 used to perform inferencing and/or training operations] FIG. 9 illustrates inference and/or training logic 915 used to perform inferencing and/or training operations associated with one or more embodiments. Details regarding inference and/or training logic 915 are provided below in conjunction with FIGS. 9 and/or 10.); configuring, using the received configuration parameters (Mallya et al. 2021/0142177 [0101 - configuration parameters] As training CNNs can be parallelized and GPU-enabled computing resources can be utilized, multiple optimization strategies can be attempted for different scenarios. A complex scenario allows tuning model architecture and preprocessing and stochastic gradient descent parameters. This expands a model configuration space. In a basic scenario, only preprocessing and stochastic gradient descent parameters are tuned. There can be a greater number of configuration parameters in a complex scenario than in a basic scenario. Tuning in a joint space can be performed using a linear or exponential number of steps, iteration through an optimization loop for models. A cost for such a tuning process can be significantly less than for tuning processes such as random search and grid search, without any significant performance loss.), the one or more inference applications (Mallya et al. 2021/0142177 [0396 - inference applications] In at least one embodiment, transfer of requests between services 4620 and inference applications may be hidden behind a software development kit (SDK), and robust transport may be provide through a queue. In at least one embodiment, a request will be placed in a queue via an API for an individual application/tenant ID combination and an SDK will pull a request from a queue and give a request to an application. In at least one embodiment, a name of a queue may be provided in an environment from where an SDK will pick it up. In at least one embodiment, asynchronous communication through a queue may be useful as it may allow any instance of an application to pick up work as it becomes available. Results may be transferred back through a queue, to ensure no data is lost. In at least one embodiment, queues may also provide an ability to segment work, as highest priority work may go to a queue with most instances of an application connected to it, while lowest priority work may go to a queue with a single instance connected to it that processes tasks in an order received. In at least one embodiment, an application may run on a GPU-accelerated instance generated in cloud 4726, and an inference service may perform inferencing on a GPU.) to process the input data (Mallya et al. 2021/0142177 [0108][0396 - inference applications…] In at least one embodiment, transfer of requests between services 4620 and inference applications may be hidden behind a software development kit (SDK), and robust transport may be provide through a queue. In at least one embodiment, a request will be placed in a queue via an API for an individual application/tenant ID combination and an SDK will pull a request from a queue and give a request to an application. In at least one embodiment, a name of a queue may be provided in an environment from where an SDK will pick it up. In at least one embodiment, asynchronous communication through a queue may be useful as it may allow any instance of an application to pick up work as it becomes available. Results may be transferred back through a queue, to ensure no data is lost. In at least one embodiment, queues may also provide an ability to segment work, as highest priority work may go to a queue with most instances of an application connected to it, while lowest priority work may go to a queue with a single instance connected to it that processes tasks in an order received. In at least one embodiment, an application may run on a GPU-accelerated instance generated in cloud 4726, and an inference service may perform inferencing on a GPU. [0079 - input data to be processed using a neural network to obtain one or more inferences or other output values, classifications, or predictions…] In at least one embodiment, at a subsequent point in time, a request may be received from client device 602 (or another such device) for content (e.g., path determinations) or data that is at least partially determined or impacted by a trained neural network. This request can include, for example, input data to be processed using a neural network to obtain one or more inferences or other output values, classifications, or predictions. In at least one embodiment, input data can be received to interface layer 608 and directed to inference module 618, although a different system or service can be used as well. In at least one embodiment, inference module 618 can obtain an appropriate trained network, such as a trained deep neural network (DNN) as discussed herein, from model repository 616 if not already stored locally to inference module 618. Inference module 618 can provide data as input to a trained network, which can then generate one or more inferences as output. This may include, for example, a classification of an instance of input data. In at least one embodiment, inferences can then be transmitted to client device 602 for display or other communication to a user. In at least one embodiment, context data for a user may also be stored to a user context data repository 622, which may include data about a user which may be useful as input to a network in generating inferences, or determining data to return to a user after obtaining instances. In at least one embodiment, relevant data, which may include at least some of input or inference data, may also be stored to a local database 620 for processing future requests. In at least one embodiment, a user can use account or other information to access resources or functionality of a provider environment. In at least one embodiment, if permitted and available, user data may also be collected and used to further train models, in order to provide more accurate inferences for future requests. In at least one embodiment, requests may be received through a user interface to a machine learning application 626 executing on client device 602, and results displayed through a same interface. A client device can include resources such as a processor 628 and memory 630 for generating a request and processing results or a response, as well as at least one data storage element 632 for storing data for machine learning application 626.); executing, on one or more processing devices, the plurality of MLMs (Mallya et al. 2021/0142177 execute machine learning models [0376; 0392; 0398]) using the one or more inference applications (Mallya et al. 2021/0142177 [0396 - inference applications] In at least one embodiment, transfer of requests between services 4620 and inference applications may be hidden behind a software development kit (SDK), and robust transport may be provide through a queue. In at least one embodiment, a request will be placed in a queue via an API for an individual application/tenant ID combination and an SDK will pull a request from a queue and give a request to an application. In at least one embodiment, a name of a queue may be provided in an environment from where an SDK will pick it up. In at least one embodiment, asynchronous communication through a queue may be useful as it may allow any instance of an application to pick up work as it becomes available. Results may be transferred back through a queue, to ensure no data is lost. In at least one embodiment, queues may also provide an ability to segment work, as highest priority work may go to a queue with most instances of an application connected to it, while lowest priority work may go to a queue with a single instance connected to it that processes tasks in an order received. In at least one embodiment, an application may run on a GPU-accelerated instance generated in cloud 4726, and an inference service may perform inferencing on a GPU.) to generate a plurality of sets of output data (Mallya et al. 2021/0142177 [0376 - execute machine learning model(s)] In at least one embodiment, where a service 4620 includes an AI service (e.g., an inference service), one or more machine learning models associated with an application for anomaly detection (e.g., tumors, growth abnormalities, scarring, etc.) may be executed by calling upon (e.g., as an API call) an inference service (e.g., an inference server) to execute machine learning model(s), or processing thereof, as part of application execution. In at least one embodiment, where another application includes one or more machine learning models for segmentation tasks, an application may call upon an inference service to execute machine learning models for performing one or more of processing operations associated with segmentation tasks. In at least one embodiment, software 4618 implementing advanced processing and inferencing pipeline that includes segmentation application and anomaly detection application may be streamlined because each application may call upon a same inference service to perform one or more inferencing tasks.), wherein each MLM of the plurality of MLMs generates at least one set of output data of the plurality of sets of output data (Mallya et al. 2021/0142177 [0079 - generate one or more inferences as output] In at least one embodiment, at a subsequent point in time, a request may be received from client device 602 (or another such device) for content (e.g., path determinations) or data that is at least partially determined or impacted by a trained neural network. This request can include, for example, input data to be processed using a neural network to obtain one or more inferences or other output values, classifications, or predictions. In at least one embodiment, input data can be received to interface layer 608 and directed to inference module 618, although a different system or service can be used as well. In at least one embodiment, inference module 618 can obtain an appropriate trained network, such as a trained deep neural network (DNN) as discussed herein, from model repository 616 if not already stored locally to inference module 618. Inference module 618 can provide data as input to a trained network, which can then generate one or more inferences as output. This may include, for example, a classification of an instance of input data. In at least one embodiment, inferences can then be transmitted to client device 602 for display or other communication to a user. In at least one embodiment, context data for a user may also be stored to a user context data repository 622, which may include data about a user which may be useful as input to a network in generating inferences, or determining data to return to a user after obtaining instances. In at least one embodiment, relevant data, which may include at least some of input or inference data, may also be stored to a local database 620 for processing future requests. In at least one embodiment, a user can use account or other information to access resources or functionality of a provider environment. In at least one embodiment, if permitted and available, user data may also be collected and used to further train models, in order to provide more accurate inferences for future requests. In at least one embodiment, requests may be received through a user interface to a machine learning application 626 executing on client device 602, and results displayed through a same interface. A client device can include resources such as a processor 628 and memory 630 for generating a request and processing results or a response, as well as at least one data storage element 632 for storing data for machine learning application 626.); and rendering, via the user API (Mallya et al. 2021/0142177 [0366 - an API may provide access to methods that allow users with appropriate credentials to associate models with applications] In at least one embodiment, model registry 4624 may be backed by object storage that may support versioning and object metadata. In at least one embodiment, object storage may be accessible through, for example, a cloud storage (e.g., cloud 4726 of FIG. 47) compatible application programming interface (API) from within a cloud platform. In at least one embodiment, machine learning models within model registry 4624 may uploaded, listed, modified, or deleted by developers or partners of a system interacting with an API. In at least one embodiment, an API may provide access to methods that allow users with appropriate credentials to associate models with applications, such that models may be executed as part of execution of containerized instantiations of applications.), a combined representation of the plurality of sets of output data (Mallya et al. 2021/0142177 [0383 - generated from computer models or renderings] In at least one embodiment, training pipelines 4704 may include AI-assisted annotation, as described in more detail herein with respect to at least FIG. 50B. In at least one embodiment, labeled clinic data 4612 (e.g., traditional annotation) may be generated by any number of techniques. In at least one embodiment, labels or other annotations may be generated within a drawing program (e.g., an annotation program), a computer aided design (CAD) program, a labeling program, another type of program suitable for generating annotations or labels for ground truth, and/or may be hand drawn, in some examples. In at least one embodiment, ground truth data may be synthetically produced (e.g., generated from computer models or renderings), real produced (e.g., designed and produced from real-world data), machine-automated (e.g., using feature analysis and learning to extract features from data and then generate labels), human annotated (e.g., labeler, or annotation expert, defines location of labels), and/or a combination thereof. In at least one embodiment, for each instance of imaging data 4608 (or other data type used by machine learning models), there may be corresponding ground truth data generated by training system 4604. In at least one embodiment, AI-assisted annotation may be performed as part of deployment pipelines 4710; either in addition to, or in lieu of AI-assisted annotation included in training pipelines 4704. In at least one embodiment, system 4700 may include a multi-layer platform that may include a software layer (e.g., software 4618) of diagnostic applications (or other application types) that may perform one or more medical imaging and diagnostic functions. In at least one embodiment, system 4700 may be communicatively coupled to (e.g., via encrypted links) PACS server networks of one or more facilities. In at least one embodiment, system 4700 may be configured to access and referenced data (e.g., DICOM data, RIS data, raw data, CIS data, REST compliant data, RPC data, raw data, etc.) from PACS servers (e.g., via a DICOM adapter 4702, or another data type adapter such as RIS, CIS, REST compliant, RPC, raw, etc.) to perform operations, such as training machine learning models, deploying machine learning models, image processing, inferencing, and/or other operations.).
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion
PERTINENT PRIOR ART – Patent Literature
The prior-art made of record and considered pertinent to applicant's disclosure.
18/229,929 – Claim 1. Zeiler et al. 2022/0405617 teaches A method comprising: receiving, via a user application programming interface (API) (Zeiler et al. 2022/0405617 [0022 - a user device 104 may interact with the platform 120 via an application programming interface (“API”)] In some embodiments, a user device 104 may interact with the platform 120 via a user interface (e.g., via a web browser) where the user interface is generated by the platform 120 and more particularly by the user interface module 114. In some embodiments, the user device 104 may be configured with an application (not shown) which allows a user to interact with the platform 120. In some embodiments, a user device 104 may interact with the platform 120 via an application programming interface (“API”) and more particularly via the interface module 118. For example, the platform 120 (or other systems associated with the platform 120) may provide one or more APIs for the submission of inputs 102 for processing by the platform 120. [0054 - user interface module] The processor 610 also communicates with a storage device 630. The storage device 630 may comprise any appropriate information storage device, including combinations of magnetic storage devices (e.g., a hard disk drive), optical storage devices, mobile telephones, and/or semiconductor memory devices. The storage device 630 stores a program 612 and/or one or more software modules 614 (e.g., associated with the user interface module, model module, threshold module, and interface module of FIG. 1) for controlling the processor 610. The processor 610 performs instructions of the programs 612, 614, and thereby operates in accordance with any of the embodiments described herein. For example, the processor 610 may receive input data and then perform processing on the input data such as described in conjunction with the processes of FIGS. 2 and 3. The programs 612, 614 may access, update and otherwise interact with data such as model data 616, collector data 618 and output data 620 as described herein.), configuration parameters (Zeiler et al. 2022/0405617 [0058 - data configurations] Although specific hardware and data configurations have been described herein, note that any number of other configurations may be provided in accordance with embodiments of the present invention (e.g., some of the information associated with the databases described herein may be combined or stored in external systems).) for execution of a plurality of machine learning models (MLMs) (Zeiler et al. 2022/0405617 [0017 - a machine learning platform 120 which receives inputs … and which produces outputs … system 100 allows one or more users operating user devices 104 to interact with the machine learning platform 120 to perform processing … machine learning platform 120 includes one or more modules that are configured to perform processing to implement and execute…] Features of some embodiments will now be described by first referring to FIG. 1 which is a block diagram of a system 100 according to some embodiments of the present invention. As shown, system 100 includes a machine learning platform 120 which receives inputs 102 (such as images, videos or the like) and which produces outputs (stored as output data 136) such as predictions and other information associated with application of models to the inputs 102. The system 100 allows one or more users operating user devices 104 to interact with the machine learning platform 120 to perform processing of those inputs 102 as described further herein. The machine learning platform 120 includes one or more modules that are configured to perform processing to implement and execute one or more “collectors” pursuant to the present invention which allow inputs from one or more sources to be collected for use in one or more models or other output destinations (such as other workflows, applications or the like).), wherein the configuration parameters identify: storage locations of the plurality of MLMs (Zeiler et al. 2022/0405617 [0052 - FIG. 4 which depicts an environment 400 in which multiple data sources 402a-n are provided and where multiple collectors 404a-n are configured to collect data from those data sources 402 and to provide output data to one or more destinations 406a-n (or data sinks)… the multiple data sources 402a-n may include multiple machine learning models constantly producing prediction streams of output data as well as other data sources (such as FTP locations, object storage locations, databases, input data streams, or the like)] In some embodiments, a collector platform 120 may be implemented in an environment in which multiple data sources are available. For example, reference is now made to FIG. 4 which depicts an environment 400 in which multiple data sources 402a-n are provided and where multiple collectors 404a-n are configured to collect data from those data sources 402 and to provide output data to one or more destinations 406a-n (or data sinks). As an example, the multiple data sources 402a-n may include multiple machine learning models constantly producing prediction streams of output data as well as other data sources (such as FTP locations, object storage locations, databases, input data streams, or the like). Embodiments of the present invention allow those rich sources of data to be filtered and otherwise processed by collectors 404 to reuse relevant items of data for use in other applications or data destinations (such as, for example, as input data for other machine learning models or applications). As shown in FIG. 4, each data source 402 may provide data to more than one collector 404 and for one or more data destination 406. As a specific illustrative example, a general model that produces an active and diverse prediction stream may provide data to a number of collectors 404 (such as a collector to collect “vehicle” data as well as a collector to collect animal or other data for different machine learning models). Embodiments allow machine learning applications to more easily obtain relevant input data for improved model development and training.), storage locations of input data into the plurality of MLMs (Zeiler et al. 2022/0405617 [0017 - inputs] Features of some embodiments will now be described by first referring to FIG. 1 which is a block diagram of a system 100 according to some embodiments of the present invention. As shown, system 100 includes a machine learning platform 120 which receives inputs 102 (such as images, videos or the like) and which produces outputs (stored as output data 136) such as predictions and other information associated with application of models to the inputs 102. The system 100 allows one or more users operating user devices 104 to interact with the machine learning platform 120 to perform processing of those inputs 102 as described further herein. The machine learning platform 120 includes one or more modules that are configured to perform processing to implement and execute one or more “collectors” pursuant to the present invention which allow inputs from one or more sources to be collected for use in one or more models or other output destinations (such as other workflows, applications or the like). [0020 - input data to ensure that only relevant or desired input data is passed from the collector to an output data sink or other destination] The present application includes a platform 120 that includes one or more collectors that are created to monitor one or more input data sources 102 (which may be, for example, prediction streams or other data sources) to collect input data therefrom for delivery to one or more output data sinks 136. For example, the input data may be obtained from a prediction stream associated with a machine learning model (such as a classification model or the like) and collector rules (including pre- and post-collector workflows as will be described below) may operate on that input data to ensure that only relevant or desired input data is passed from the collector to an output data sink or other destination. [0021 - allow the selection of specific input data for output to the output locations…] According to some embodiments, the platform 120 may include one or more “automated” collectors that may automatically receive or monitor input data from one or more data sources, perform processing on that data, and output the data to one or more data sinks or output destinations. The processing may allow the selection of specific input data for output to the output locations, allowing complex operations to be performed to ensure appropriate data is output. The result is a system that ensures accurate data is presented at an output without manual intervention or further processing outside the system 100.), and one or more inference applications for performing inference processing of the input data using the plurality of MLMs; configuring, using the received configuration parameters, the one or more inference applications to process the input data; executing, on one or more processing devices (Zeiler et al. 2022/0405617 [0017 - machine learning platform 120 includes one or more modules that are configured to perform processing to implement and execute…] Features of some embodiments will now be described by first referring to FIG. 1 which is a block diagram of a system 100 according to some embodiments of the present invention. As shown, system 100 includes a machine learning platform 120 which receives inputs 102 (such as images, videos or the like) and which produces outputs (stored as output data 136) such as predictions and other information associated with application of models to the inputs 102. The system 100 allows one or more users operating user devices 104 to interact with the machine learning platform 120 to perform processing of those inputs 102 as described further herein. The machine learning platform 120 includes one or more modules that are configured to perform processing to implement and execute one or more “collectors” pursuant to the present invention which allow inputs from one or more sources to be collected for use in one or more models or other output destinations (such as other workflows, applications or the like).), the plurality of MLMs using the one or more inference applications to generate a plurality of sets of output data (Zeiler et al. 2022/0405617 [0018 - models that are configured to receive and process inputs 102 and generate output…] The system 100 may generally be referred to herein as being (or as a part of) a “machine learning system”. The system 100 can include one or more models that may be stored at model database 132 and interacted with via a component or controller such as model module 112. In some embodiments, one or more of the models may be so-called “classification” models that are configured to receive and process inputs 102 and generate output data 136. As used herein, the term “classification model” can include various machine learning models, including but not limited to a “detection model” or a “regression model.” Embodiments may be used with other models, and the use of a classification model as the illustrative example is intended to be illustrative but not limiting. As a result, the term “model” as used herein, is used to refer to any of a number of different types of models (from classification models to segmentation models or the like). As used herein, the term “classification model” can include various machine learning models, including but not limited to a “detection model” or a “regression model.” Embodiments may be used with other models, and the use of a classification model as the illustrative example is intended to be illustrative but not limiting. As a result, the term “model” as used herein, is used to refer to any of a number of different types of models (from classification models to segmentation models or the like).), wherein each MLM of the plurality of MLMs generates at least one set of output data of the plurality of sets of output data (Zeiler et al. 2022/0405617 [0017 - machine learning platform 120 which receives inputs … and which produces outputs…] Features of some embodiments will now be described by first referring to FIG. 1 which is a block diagram of a system 100 according to some embodiments of the present invention. As shown, system 100 includes a machine learning platform 120 which receives inputs 102 (such as images, videos or the like) and which produces outputs (stored as output data 136) such as predictions and other information associated with application of models to the inputs 102. The system 100 allows one or more users operating user devices 104 to interact with the machine learning platform 120 to perform processing of those inputs 102 as described further herein. The machine learning platform 120 includes one or more modules that are configured to perform processing to implement and execute one or more “collectors” pursuant to the present invention which allow inputs from one or more sources to be collected for use in one or more models or other output destinations (such as other workflows, applications or the like). [0018 - models that are configured to receive and process inputs 102 and generate output data…] The system 100 may generally be referred to herein as being (or as a part of) a “machine learning system”. The system 100 can include one or more models that may be stored at model database 132 and interacted with via a component or controller such as model module 112. In some embodiments, one or more of the models may be so-called “classification” models that are configured to receive and process inputs 102 and generate output data 136. As used herein, the term “classification model” can include various machine learning models, including but not limited to a “detection model” or a “regression model.” Embodiments may be used with other models, and the use of a classification model as the illustrative example is intended to be illustrative but not limiting. As a result, the term “model” as used herein, is used to refer to any of a number of different types of models (from classification models to segmentation models or the like). As used herein, the term “classification model” can include various machine learning models, including but not limited to a “detection model” or a “regression model.” Embodiments may be used with other models, and the use of a classification model as the illustrative example is intended to be illustrative but not limiting. As a result, the term “model” as used herein, is used to refer to any of a number of different types of models (from classification models to segmentation models or the like). [0020 - input data may be obtained … associated with a machine learning model … and collector rules … may operate on that input data to ensure that only relevant or desired input data is passed from the collector to an output data sink…] The present application includes a platform 120 that includes one or more collectors that are created to monitor one or more input data sources 102 (which may be, for example, prediction streams or other data sources) to collect input data therefrom for delivery to one or more output data sinks 136. For example, the input data may be obtained from a prediction stream associated with a machine learning model (such as a classification model or the like) and collector rules (including pre- and post-collector workflows as will be described below) may operate on that input data to ensure that only relevant or desired input data is passed from the collector to an output data sink or other destination.); and rendering, via the user API, a combined representation of the plurality of sets of output data (Zeiler et al. 2022/0405617 [0053 - a computer monitor to display reports and results] The embodiments described herein may be implemented using any number of different hardware configurations. For example, FIG. 6 illustrates a collector platform 600 that may be, for example, associated with the system 100 of FIG. 1 as well as the other systems and components described herein. The collector platform 600 comprises a processor 610, such as one or more commercially available central processing units (CPUs) in the form of microprocessors, coupled to a communication device 620 configured to communicate via a communication network (not shown in FIG. 6). The communication device 620 may be used to communicate, for example, with one or more input sources and/or user devices. The collector platform 600 further includes an input device 640 (e.g., a mouse and/or keyboard to define rules and relationships) and an output device 650 (e.g., a computer monitor to display reports and results to an administrator).).
Wetherbee et al. 2021/0174248 [0056 - generating a recommended configuration of central processing unit(s), graphics processing unit(s), and memory for the target cloud environment to execute the machine learning application]
Li et al. 2023/0043584 [0032 - dynamic integer format optimization may be performed by run-time MOE 138 on edge computing device 130. The deployed and optimized MLMs 108 may be used by inference engine 150 to process application-specific (inference) data 152 and produce inference output]
Geigel 2018/0329740 [Abstract - distributed machine learning engine is proposed that allows for optimization and parallel execution of the machine learning tasks]
Yalla et al. 2022/0076145 [0066 - execute inference of machine learning models in parallel]
Boue et al. 2024/0370766 [0015 - machine learning model during a retraining phase, which can execute concurrently (e.g., in parallel) with inference phases of the machine learning model or in batch mode (e.g., in sequence with different inference phases of the machine learning mode]
PERTINENT PRIOR ART – Non-Patent Literature (NPL)
The NPL prior-art made of record and considered pertinent to applicant's disclosure.
Ranking and automatic selection of machine learning models, An IP.com Prior Art Database Technical Disclosure, Authors et. al.: Disclosed Without Attribution, IP.com Number: IPCOM000252275D, IP.com Electronic Publication Date: January 03, 2018
THIS ACTION IS MADE FINAL
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
THIS ACTION IS MADE FINAL
Applicant’s amendment necessitated new grounds of rejection and FINAL Rejection.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW T. SITTNER whose telephone number is (571) 270-7137 and email: matthew.sittner@uspto.gov. The examiner can normally be reached on Monday-Friday, 8:00am - 5:00pm (Mountain Time Zone). Please schedule interview requests via email: matthew.sittner@uspto.gov
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sarah M. Monfeldt can be reached on (571) 270-1833.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW T SITTNER/
Primary Examiner, Art Unit 3629b
Prosecution Insights