DETAILED ACTION
Notice of AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s Amendment and remarks dated 9/14/2026 have been considered. Claims 1-20 are pending.
Drawing Objections. The objections to Figs. 1 and 6A are withdrawn in view of the substitute drawings submitted by Applicant.
Specification Objections. The objection to the specification is withdrawn in view of Applicant’s amendments to the specification.
Response to Arguments
On page 9 of Applicant’s 9/14/2026 Amendment and remarks, Applicant asserts that no new matter has been added via the amendments.
After review, the examiner agrees that no new matter has been added via the amendments. The amendments to the independent claims are supported by the original versions of the independent claims themselves and para. 0053 of the instant specification. The amendments to the dependent claims are minor and merely clarify claim language based on the amendments to the independent claims.
On pages 10-11 of Applicant’s 9/14/2026 Amendment and remarks, with respect to the rejections under 35 U.S.C. 101, Applicant argues that as amended, the claims no longer recite any mental processes under Step 2A, Prong 1, and therefore the rejections should be withdrawn.
Applicant’s amendments and arguments are persuasive. All rejections under 35 U.S.C. 101 are withdrawn because the independent claims no longer recite any mental processes under Step 2A, prong 1. The examiner respectfully submits that Applicant’s remaining arguments with respect to 35 U.S.C. 101 are moot in view of the withdrawal of all such rejections.
On pages 14-15 of Applicant’s 9/14/2026 Amendment and remarks, with respect to the rejections under 35 U.S.C. 103, Applicant argues that the GOLI reference only recites adjusting “runtime environment settings” and that such disclosure allegedly does not disclose the “one or more runtime settings of a machine learning model (MLM)” limitation.
The examiner respectfully disagrees. Significantly, Applicant has ignored the disclosure of para. 0028 of GOLI, which recites: “Runtime information for the ML model may be determined based on heuristics and statistics collected for similar ML models, which can be estimated based on size.” As such, para. 0028 clearly discloses that runtime information for the ML model (which is part of the runtime environment” is determined (e.g., determined so that it can be configured). Therefore, GOLI expressly teaches this limitation.
Moreover, the examiner respectfully submits that under the broadest reasonable interpretation of the ““one or more runtime settings of a machine learning model (MLM)” limitation, that the more general disclosures of GOLI related to the respective runtime environment settings and memory usage teach this limitation. The specification and claims do not provide a specific definition for “one or more runtime settings of a machine learning model (MLM)”, and in view of the explanation at para. 0054 of the instant specification, the broadest reasonable interpretation of “adjusting the runtime settings of the MLM” includes adjusting “hardware settings for one or more processing platforms used by the MLM”, and GOLI teaches that runtime environment settings (corresponding to such hardware settings) can be adjusted.
On pages 15-16 of Applicant’s 9/14/2026 Amendment and remarks, with respect to the rejections under 35 U.S.C. 103, Applicant argues that the “based on the detected watermark data of LUO” statement is not supported by GOLI.
The examiner does not dispute that GOLI does not make reference to LUO. However, that statement was made on page 42 of the 5/15/2026 office action as an example about how one of ordinary skill in the art could have modified the end-to-end watermark system of LUO using the teachings of adjusting a ML models runtime settings. This is a hypothetical combination where the office action explains one possible combination of the references, and specific “bridging statements” are not required.
On pages 16-17 of Applicant’s 9/14/2026 Amendment and remarks, with respect to the rejection of claims 7-8. 14-15, and 20 under 35 U.S.C. 103, Applicant argues that the DESAI reference only applies to “offline training, not runtime setting adjustment.”
The examiner respectfully disagrees. In response to Applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). As discussed above, GOLI clearly teaches adjusting runtime settings, and now the combination of LUO, GOLI, and DESAI uses the loss function teachings of DESAI to change the runtime settings as taught by GOLI.
On pages 17 of Applicant’s 9/14/2026 Amendment and remarks, with respect to the rejection of claims 8 and 15 under 35 U.S.C. 103, Applicant argues that the prior art of record does not teach a second machine learning model as an “evaluation MLM.”
Applicant’s arguments are persuasive. The rejections to claims 8 and 15 under 35 U.S.C. 103 are withdrawn.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-4, 6, 9-13, and 16-19 are rejected under 35 U.S.C. 103 as being unpatentable over US 20230362399 A1, hereinafter referenced as LUO, in view of US 20200356415 A1, hereinafter referenced as GOLI.
Regarding Claim 1
LUO teaches:
A method comprising: (LUO, para. 0004: “In general, one innovative aspect of the subject matter described in this specification can be embodied in methods including the operations for jointly training an encoder machine learning model that generates a digital watermark that is embedded into an image”)
augmenting, using one or more processing units, one or more data streams with auxiliary data to obtain an augmented data stream; and (LUO, para. 0028: “The visually discernible watermarks, referred to as simply “watermarks” for brevity, are semi-transparent, and visually discernible to a human user under normal viewing conditions, such that the watermarks can be embedded in content without degrading the visual quality of the content. The watermarks can carry information, such as an identifier of a source of the images in which they are embedded. For example, in the context of the Internet, a watermark can identify (among other information) an entity, server, or service that placed the content on a publisher's property (e.g., website, video stream, video game, or mobile application) when the publisher's property was accessed by a user. As such, when a reproduction of the content (e.g., a picture or screenshot of the content), as presented on the publisher's property, is captured and submitted for verification, the watermark can be detected and decoded to verify whether the content was, in fact, distributed by the appropriate entity, server, or service.”;
LUO, para. 0036: “For purposes of example, the description that follows are explained with reference to source images 128a-n that are provided to the client device 104, but it should be appreciated that second watermark 126 can be overlaid on various other types of visible content, including native application content, streaming video content, video game content, or other visible content. It should also be noted that instead of augmenting the response served to the client device 104 with a semi-transparent second watermark 126, the server system 102 can deliver the watermarks to the one or more content providers 106a-n where the content providers can generate a watermarked image prior to transmitting content to the client device 104 for presentation.”
LUO, para. 0036: “In such implementations, the watermark generator 110 can be implemented by the content providers 106a-n that generates semi-transparent watermark so that the content providers 106a-n has access to semi-transparent watermark to generate an encoding image 130 when responding to the request from the client device 104.”
LUO, para. 0136: “FIG. 8 is a block diagram of an example computer system 800 that can be used to perform operations described above. The system 800 includes a processor 810, a memory 820, a storage device 830, and an input/output device 840. Each of the components 810, 820, 830, and 840 can be interconnected, for example, using a system bus 850. The processor 810 is capable of processing instructions for execution within the system 800. In some implementations, the processor 810 is a single-threaded processor. In another implementation, the processor 810 is a multi-threaded processor. The processor 810 is capable of processing instructions stored in the memory 820 or on the storage device 830.”’;
Examiner’s Note: LUO discloses embedding images, which can be part of a streaming video stream (corresponding to recited “one or more data streams”) with a semi-transparent watermark (corresponding to recited “auxiliary data”) to create a watermarked video stream)
… a characterization, generated by the MLM from the augmented data stream, of a presence of the auxiliary data in the augmented data stream (LUO, para. 0058: “In some implementations, the watermark and distortion detection apparatus 132 can implement a watermark detector machine learning model 132a that is configured to process the possibly encoded image 302 and generate, as output, an indication of whether a portion of the possibly encoded image 302 includes one or more watermarks. The watermark detector machine learning model 132a can be any model deemed suitable for the specific implementation, such as decision trees, artificial neural networks, genetic programming, logic programming, support vector machines, clustering, reinforcement learning, Bayesian inferencing, etc. Machine learning models may also include methods, algorithms and techniques for computer vision and image processing for analyzing images.”
LUO, para. 0062: “FIG. 3B is a block diagram of an example architecture watermark detector machine learning model 350. The watermark detector machine learning model 350 is a CNN with UNet architecture.”;
Examiner’s Note: LUO discloses a watermark detector machine learning model, that can analyze an image (part of the watermarked video stream) to obtain an indicator of whether the image includes one or more watermarks (corresponding to recited “a characterization, generated by the MLM from the augmented data stream, of a presence of the auxiliary data in the augmented data stream”))
However, LUO fails to explicitly teach:
adjusting, by the one or more processing units, one or more runtime settings of a machine learning model (MLM) based at least in part on a characterization
However, in a related field of endeavor (machine learning inferencing in edge systems, including systems for video processing, see para. 0013), GOLI teaches and make obvious:
adjusting, by the one or more processing units, one or more runtime settings of a machine learning model (MLM) based at least in part on a characterization (GOLI, para. 0013: “ In other examples, the ML inference generation tool may configure the ML model application to allow each edge system to which the ML model application is deployed to choose an execution path that uses respective runtime environment settings and memory usage corresponding to specialized hardware and other hardware configurations of the respective edge system.”;
GOLI, para. 0028: “The independent generation of each ML model application by the ML model application generator 144, may include choosing respective runtime environment settings and memory usage based on specialized hardware (e.g., GPU, TPU, hardware accelerators, VPU, Movidius, etc.) and other hardware configurations of the edge system. Runtime information for the ML model may be determined based on heuristics and statistics collected for similar ML models, which can be estimated based on size.”;
Examiner’s Note: GOLI teaches that a machine learning application, running on a respective edge system, can select and use different runtime settings, where such settings can be based on collected information, and where “Runtime information for the ML model may be determined”; the LUO-GOLI combination now modifies the end-to-end watermark system of LUO to operate with respect to edge devices that stream video as disclosed by GOLI, such that the ML models running on such edge devices can adjust their runtime settings as disclosed by GOLI, based on the detected watermark data of LUO)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of LUO with GOLI as explained above. As disclosed by GOLI, one of ordinary skill would have been motivated to do so in order to “improve efficiency in executing the ML model at the edge system as compared with generic or non-specific ML model applications.” (para. 0079). One of ordinary skill in the art would further understand the benefit of using watermarking in video for anti-piracy purposes, for example, to stop streaming video if it contains a certain type of watermark.
Regarding Claim 2
LUO and GOLI teach the method of claim 1 as explained above. LUO further teaches:
wherein the characterization comprises at least one of:
a missed presence of the auxiliary data in the augmented data stream;
a location of the auxiliary data in the augmented data stream;
a type of the auxiliary data in the augmented data stream;
a confidence in detecting the location of the auxiliary data;
a confidence in detecting the type of the auxiliary data. (LUO, para. 0058: “The watermark detector machine learning model 132a can be any model deemed suitable for the specific implementation, such as decision trees, artificial neural networks, genetic programming, logic programming, support vector machines, clustering, reinforcement learning, Bayesian inferencing, etc. Machine learning models may also include methods, algorithms and techniques for computer vision and image processing for analyzing images. In such implementations, the indication of whether the possibly encoded image 302 includes a portion of a watermark or one or more watermarks can be of the form of a classification or a number such as a score or a probability. For example, the watermark detector machine learning model 132a can be implemented as a classification model that can process the possibly encoded image 302 to classify the image as an image that includes a watermark or an image that does not include a watermark. In another example, the watermark detector machine learning model 132a can process the possibly encoded image 302 to generate a score such as a score that indicates a likelihood that the possibly encoded image 302 includes a watermark.”;
Examiner’s Note: the broadest reasonable interpretation of this limitation only requires a single one of the alternatives to be met, and LUO teaches determining a likelihood that the image includes a watermark, corresponding to “a confidence in detecting the type of the auxiliary data”, where the type of the auxiliary data is a watermark, and the confidence relates to the probability score)
Regarding Claim 3
LUO and GOLI teach the method of claim 1 as explained above. LUO further teaches:
wherein the one or more data streams comprise one or more video streams. (LUO, para. 0036: “For purposes of example, the description that follows are explained with reference to source images 128a-n that are provided to the client device 104, but it should be appreciated that second watermark 126 can be overlaid on various other types of visible content, including native application content, streaming video content, video game content, or other visible content. It should also be noted that instead of augmenting the response served to the client device 104 with a semi-transparent second watermark 126, the server system 102 can deliver the watermarks to the one or more content providers 106a-n where the content providers can generate a watermarked image prior to transmitting content to the client device 104 for presentation.”)
Regarding Claim 4
LUO and GOLI teach the method of claim 1 as explained above. LUO further teaches:
wherein the auxiliary data comprises an image of an object external to the one or more data streams. (LUO, para. 0028: “The visually discernible watermarks, referred to as simply “watermarks” for brevity, are semi-transparent, and visually discernible to a human user under normal viewing conditions, such that the watermarks can be embedded in content without degrading the visual quality of the content. The watermarks can carry information, such as an identifier of a source of the images in which they are embedded. For example, in the context of the Internet, a watermark can identify (among other information) an entity, server, or service that placed the content on a publisher's property (e.g., website, video stream, video game, or mobile application) when the publisher's property was accessed by a user. As such, when a reproduction of the content (e.g., a picture or screenshot of the content), as presented on the publisher's property, is captured and submitted for verification, the watermark can be detected and decoded to verify whether the content was, in fact, distributed by the appropriate entity, server, or service.”;
Examiner’s Note: the inserted watermark is a new, visually discernable mark, which is not native to the image or streaming video (corresponding to recited “image of an object external to the one or more data streams”))
Regarding Claim 6
LUO and GOLI teach the method of claim 1 as explained above. However, LUO fails to explicitly teach:
wherein the one or more runtime settings of the MLM comprise:
one or more settings for rescaling portions of data of the one or more data streams;
one or more settings for a clustering algorithm used by the MLM to process the one or more data streams;
a type of a codec used to encode data in the one or more data streams;
a bitrate of the one or more data streams; or
one or more hardware settings for one or more processing platforms used by the MLM to process the one or more data streams.
However, in a related field of endeavor (machine learning inferencing in edge systems, including systems for video processing, see para. 0013), GOLI teaches and make obvious:
wherein the one or more runtime settings of the MLM comprise:
one or more settings for rescaling portions of data of the one or more data streams;
one or more settings for a clustering algorithm used by the MLM to process the one or more data streams;
a type of a codec used to encode data in the one or more data streams;
a bitrate of the one or more data streams; or
one or more hardware settings for one or more processing platforms used by the MLM to process the one or more data streams. (GOLI, para. 0013: “ In other examples, the ML inference generation tool may configure the ML model application to allow each edge system to which the ML model application is deployed to choose an execution path that uses respective runtime environment settings and memory usage corresponding to specialized hardware and other hardware configurations of the respective edge system.”;
GOLI, para. 0028: “The independent generation of each ML model application by the ML model application generator 144, may include choosing respective runtime environment settings and memory usage based on specialized hardware (e.g., GPU, TPU, hardware accelerators, VPU, Movidius, etc.) and other hardware configurations of the edge system. Runtime information for the ML model may be determined based on heuristics and statistics collected for similar ML models, which can be estimated based on size.”;
Examiner’s Note: the broadest reasonable interpretation of this limitation only requires a single one of the alternatives to be met, and GOLI teaches changing “one or more hardware settings for one or more processing platforms used by the MLM to process the one or more data streams”; the LUO-GOLI combination now modifies the end-to-end watermark system of LUO to operate with respect to edge devices that stream video as disclosed by GOLI, such that the ML models running on such edge devices can adjust their runtime settings with respect to particular hardware settings as disclosed by GOLI)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of LUO with GOLI as explained above. As disclosed by GOLI, one of ordinary skill would have been motivated to do so in order to “improve efficiency in executing the ML model at the edge system as compared with generic or non-specific ML model applications.” (para. 0079). One of ordinary skill in the art would further understand the benefit of using watermarking in video for anti-piracy purposes, for example, to stop streaming video if it contains a certain type of watermark.
Regarding Claim 9
LUO and GOLI teach the method of claim 1 as explained above. However, LUO fails to explicitly teach:
wherein adjusting the one or more runtime settings of the MLM comprises: modifying an execution of the MLM on one or more computational resources.
However, in a related field of endeavor (machine learning inferencing in edge systems, including systems for video processing, see para. 0013), GOLI teaches and make obvious:
wherein adjusting the one or more runtime settings of the MLM comprises: modifying an execution of the MLM on one or more computational resources. (GOLI, para. 0013: “ In other examples, the ML inference generation tool may configure the ML model application to allow each edge system to which the ML model application is deployed to choose an execution path that uses respective runtime environment settings and memory usage corresponding to specialized hardware and other hardware configurations of the respective edge system.”;
GOLI, para. 0028: “The independent generation of each ML model application by the ML model application generator 144, may include choosing respective runtime environment settings and memory usage based on specialized hardware (e.g., GPU, TPU, hardware accelerators, VPU, Movidius, etc.) and other hardware configurations of the edge system. Runtime information for the ML model may be determined based on heuristics and statistics collected for similar ML models, which can be estimated based on size.”;
Examiner’s Note: the LUO-GOLI combination now modifies the end-to-end watermark system of LUO to operate with respect to edge devices that stream video as disclosed by GOLI, such that the ML models running on such edge devices can adjust their runtime settings with respect to the particular hardware and computational resources available as disclosed by GOLI)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of LUO with GOLI as explained above. As disclosed by GOLI, one of ordinary skill would have been motivated to do so in order to “improve efficiency in executing the ML model at the edge system as compared with generic or non-specific ML model applications.” (para. 0079). One of ordinary skill in the art would further understand the benefit of using watermarking in video for anti-piracy purposes, for example, to stop streaming video if it contains a certain type of watermark.
Claim 10 recites a system, utilizing a processing device, that corresponds to the method of claim 1, and is therefore rejected for the same reasons explained above with respect to claim 1.
Claim 11 depends from claim 10 and claims a system that corresponds to the method of claim 2, and is therefore rejected for the same reasons explained above with respect to claims 2 and 10.
Claim 12 depends from claim 10 and claims a system that corresponds to the method of claim 3, and is therefore rejected for the same reasons explained above with respect to claims 3 and 10.
Claim 13 depends from claim 10 and claims a system that corresponds to the method of claim 6, and is therefore rejected for the same reasons explained above with respect to claims 6 and 10.
Claim 15 depends from claim 10 and claims a system that corresponds to the method of claim 8, and is therefore rejected for the same reasons explained above with respect to claims 8 and 10.
Claim 16 depends from claim 10 and claims a system that corresponds to the method of claim 9, and is therefore rejected for the same reasons explained above with respect to claims 9 and 10.
Regarding Claim 17
LUO and GOLI teach the method of claim 10 as explained above. LUO further teaches:
wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations; (LUO, para. 0100: “The server system 102 applies distortions to the watermarked image (525). As described with reference to FIG. 4, the distortion apparatus 440 can process the watermarked training image 430 to generate one or more distorted images 450. The distorted images 450 are generated by adding one or more distortions such as vertical and horizontal scaling, splicing with other background images, JPEG compression, cropping to simulate real world image alteration that a possibly encoded image 302 in FIG. 3 can undergo. For example, the distortion apparatus 440 can distort the watermarked training image 430 based on a random vertical scaling factor.”)
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations; (LUO, para. 0061: “A deep convolutional neural network (CNN) with a UNet architecture is further explained with reference to FIG. 3B that can be used as a watermark detector machine learning model 132a.”)
a system implemented using an edge device;
a system for generating or presenting at least one of augmented reality content, virtual reality content, or mixed reality content; (LUO, para. 0031: “ In some implementations, the client device 104 includes an electronic display device (e.g., an LCD or LED screen, a CRT monitor, a head-mounted virtual reality display, a head-mounted mixed-reality display), or is coupled to an electronic display device, that displays content from the rendered response to the one or more users of the client device 104. “)
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data using AI operations;
a system incorporating one or more virtual machines (VMs); (LUO, para. 0143: “The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them.”)
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources. (LUO, para. 0138: “The storage device 830 is capable of providing mass storage for the system 800. In some implementations, the storage device 830 is a computer-readable medium. In various different implementations, the storage device 830 can include, for example, a hard disk device, an optical disk device, a storage device that is shared over a network by multiple computing devices (e.g., a cloud storage device), or some other large capacity storage device.”)
Examiner’s Note: the broadest reasonable interpretation of this limitation only requires a single one of the alternatives to be met, and LUO teaches at least the limitations indicated above)
Regarding Claim 18
LUO teaches:
A system comprising:
one or more processing units to: augment the one or more data streams with auxiliary data to obtain an augmented data stream; (LUO, para. 0028: “The visually discernible watermarks, referred to as simply “watermarks” for brevity, are semi-transparent, and visually discernible to a human user under normal viewing conditions, such that the watermarks can be embedded in content without degrading the visual quality of the content. The watermarks can carry information, such as an identifier of a source of the images in which they are embedded. For example, in the context of the Internet, a watermark can identify (among other information) an entity, server, or service that placed the content on a publisher's property (e.g., website, video stream, video game, or mobile application) when the publisher's property was accessed by a user. As such, when a reproduction of the content (e.g., a picture or screenshot of the content), as presented on the publisher's property, is captured and submitted for verification, the watermark can be detected and decoded to verify whether the content was, in fact, distributed by the appropriate entity, server, or service.”;
LUO, para. 0036: “For purposes of example, the description that follows are explained with reference to source images 128a-n that are provided to the client device 104, but it should be appreciated that second watermark 126 can be overlaid on various other types of visible content, including native application content, streaming video content, video game content, or other visible content. It should also be noted that instead of augmenting the response served to the client device 104 with a semi-transparent second watermark 126, the server system 102 can deliver the watermarks to the one or more content providers 106a-n where the content providers can generate a watermarked image prior to transmitting content to the client device 104 for presentation.”
LUO, para. 0036: “In such implementations, the watermark generator 110 can be implemented by the content providers 106a-n that generates semi-transparent watermark so that the content providers 106a-n has access to semi-transparent watermark to generate an encoding image 130 when responding to the request from the client device 104.”
LUO, para. 0136: “FIG. 8 is a block diagram of an example computer system 800 that can be used to perform operations described above. The system 800 includes a processor 810, a memory 820, a storage device 830, and an input/output device 840. Each of the components 810, 820, 830, and 840 can be interconnected, for example, using a system bus 850. The processor 810 is capable of processing instructions for execution within the system 800. In some implementations, the processor 810 is a single-threaded processor. In another implementation, the processor 810 is a multi-threaded processor. The processor 810 is capable of processing instructions stored in the memory 820 or on the storage device 830.”’;
Examiner’s Note: LUO discloses embedding images, which can be part of a streaming video stream (corresponding to recited “one or more data streams”) with a semi-transparent watermark (corresponding to recited “auxiliary data”) to create a watermarked video stream)
… a characterization, generated by the MLM from the augmented data stream, of a presence of the auxiliary data in the augmented data stream (LUO, para. 0058: “In some implementations, the watermark and distortion detection apparatus 132 can implement a watermark detector machine learning model 132a that is configured to process the possibly encoded image 302 and generate, as output, an indication of whether a portion of the possibly encoded image 302 includes one or more watermarks. The watermark detector machine learning model 132a can be any model deemed suitable for the specific implementation, such as decision trees, artificial neural networks, genetic programming, logic programming, support vector machines, clustering, reinforcement learning, Bayesian inferencing, etc. Machine learning models may also include methods, algorithms and techniques for computer vision and image processing for analyzing images.”
LUO, para. 0062: “FIG. 3B is a block diagram of an example architecture watermark detector machine learning model 350. The watermark detector machine learning model 350 is a CNN with UNet architecture.”;
Examiner’s Note: LUO discloses a watermark detector machine learning model, that can analyze an image (part of the watermarked video stream) to obtain an indicator of whether the image includes one or more watermarks (corresponding to recited “a characterization, generated by the MLM from the augmented data stream, of a presence of the auxiliary data in the augmented data stream”))
However, LUO fails to explicitly teach:
one or more sensors to generate one or more data streams; and
adjust one or more runtime settings of a machine learning model (MLM) based at least in part on a characterization
However, in a related field of endeavor (machine learning inferencing in edge systems, including systems for video processing, see para. 0013), GOLI teaches and make obvious:
one or more sensors to generate one or more data streams; and (GOLI, para. 0019: “The data sources may include sensors (e.g., electrical, temperature, matter flow, movement, position, biometric data, or any other type of sensor), cameras, transducers, any type of RF receiver, or any other type of device configured to receive and/or generate source data.”;
Examiner’s Note: the LUO-GOLI combination now modifies the end-to-end watermark system of LUO to operate with respect to edge devices to use the camera of GOLI to detect streaming video data)
adjust one or more runtime settings of a machine learning model (MLM) based at least in part on a characterization (GOLI, para. 0013: “ In other examples, the ML inference generation tool may configure the ML model application to allow each edge system to which the ML model application is deployed to choose an execution path that uses respective runtime environment settings and memory usage corresponding to specialized hardware and other hardware configurations of the respective edge system.”;
GOLI, para. 0028: “The independent generation of each ML model application by the ML model application generator 144, may include choosing respective runtime environment settings and memory usage based on specialized hardware (e.g., GPU, TPU, hardware accelerators, VPU, Movidius, etc.) and other hardware configurations of the edge system. Runtime information for the ML model may be determined based on heuristics and statistics collected for similar ML models, which can be estimated based on size.”;
Examiner’s Note: GOLI teaches that a machine learning application, running on a respective edge system, can select and use different runtime settings, where such settings can be based on collected information, and where and where “Runtime information for the ML model may be determined”; the LUO-GOLI combination now modifies the end-to-end watermark system of LUO to operate with respect to edge devices that stream video as disclosed by GOLI, such that the ML models running on such edge devices can adjust their runtime settings as disclosed by GOLI, based on the detected watermark data of LUO)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of LUO with GOLI as explained above. As disclosed by GOLI, one of ordinary skill would have been motivated to do so in order to “improve efficiency in executing the ML model at the edge system as compared with generic or non-specific ML model applications.” (para. 0079). One of ordinary skill in the art would further understand the benefit of using watermarking in video for anti-piracy purposes, for example, to stop streaming video if it contains a certain type of watermark.
Claim 19 depends from claim 18 and claims a system that corresponds to the method of claim 2, and is therefore rejected for the same reasons explained above with respect to claims 2 and 18.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable LUO in view of GOLI and further in view of US 20210049800 A1, hereinafter referenced as NASH.
Regarding Claim 5
LUO and GOLI teach the method of claim 1 as explained above. However, LUO and GOLI fail to explicitly teach:
wherein augmenting the one or more data streams with the auxiliary data comprises: blending the auxiliary data into at least one stream of the one or more data streams.
However, in a related field of endeavor (adding content to images, see para. 0001), NASH teaches and makes obvious:
wherein augmenting the one or more data streams with the auxiliary data comprises: blending the auxiliary data into at least one stream of the one or more data streams. (NASH, para. 0027: “Also, a similar functionality may be implemented that allows content (e.g., advertisements, logos, watermarks, etc.) to be “blended” with the selected portions of the images. For example, if the content includes a logo, the color of a portion of the logo may be adjusted to “blend in” with the selected portions of the images (e.g., so that the logo is still visible but not overly distracting). Such features may be implemented by analyzing the portions of the images (and/or the content) using computing vision techniques and/or a cognitive analysis.”;
Examiner’s Note: the LUO-GOLI-NASH combination now modifies the end-to-end watermarking system of LUO to embed the watermark using the blending techniques of NASH so that the watermark is visible but not overly distracting)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of LUO with GOLI and NASH as explained above. As disclosed by NASH, one of ordinary skill would have been motivated to do so in order to implement watermarks in a way so that they are “still visible but not overly distracting.” (para. 0027).
Claims 7, 14, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over LUO in view of GOLI and further in view of US 20230342677 A1, hereinafter referenced as DESAI.
Regarding Claim 7
LUO and GOLI teach the method of claim 1 as explained above. However, LUO and GOLI fail to explicitly teach:
wherein adjusting the one or more runtime settings of the MLM comprises: applying, using a ground truth for the auxiliary data, a loss function to the characterization to compute a loss value characterizing accuracy of the characterization of the presence of the auxiliary data in the augmented data stream; and
modifying the one or more runtime settings of the MLM to reduce the loss value.
However, in a related field of endeavor (an augmented reality environment where virtual objects are superimposed over a physical environment, see para. 0044), DESAI teaches and makes obvious:
wherein adjusting the one or more runtime settings of the MLM comprises: applying, using a ground truth for the auxiliary data, a loss function to the characterization to compute a loss value characterizing accuracy of the characterization of the presence of the auxiliary data in the augmented data stream; and (DESAI, para. 0146: “In some techniques, as part of deriving the hypothesis function, a cost or loss function 940 may be defined that measures the difference between the ground truth value for an input and the predicted value for that input. The delta between the ground truth value for an input and the predicted value for that input may be used to update the parameters of the one or more computer vision models 915. The update of the parameters may be performed using techniques such as back propagation, random feedback, Direct Feedback Alignment (DFA), Indirect Feedback Alignment (IFA), Hebbian learning, and the like to minimize this cost or loss function 940. After the one or more computer vision models 915 has been trained, the one or more computer vision models 915 may then be stored in a model store where the model can be executed for inferencing or making predictions during the inference or runtime phase based upon real time or inferring data points.”;
Examiner’s Note: the LUO-GOLI-DESAI combination now modifies the end-to-end watermarking system to utilize the loss function of DESAI, such that a loss is determined between the ground truth and the predicted value (a watermark) with respect to DESAI, and used to adjust the settings of the machine learning model that can be applied at runtime).
modifying the one or more runtime settings of the MLM to reduce the loss value. (DESAI, para. 0146: “In some techniques, as part of deriving the hypothesis function, a cost or loss function 940 may be defined that measures the difference between the ground truth value for an input and the predicted value for that input. The delta between the ground truth value for an input and the predicted value for that input may be used to update the parameters of the one or more computer vision models 915. The update of the parameters may be performed using techniques such as back propagation, random feedback, Direct Feedback Alignment (DFA), Indirect Feedback Alignment (IFA), Hebbian learning, and the like to minimize this cost or loss function 940. After the one or more computer vision models 915 has been trained, the one or more computer vision models 915 may then be stored in a model store where the model can be executed for inferencing or making predictions during the inference or runtime phase based upon real time or inferring data points.”;
Examiner’s Note: the LUO-GOLI-DESAI combination now modifies the end-to-end watermarking system to utilize the loss function of DESAI, such that a loss is determined between the ground truth and the predicted value (a watermark) with respect to DESAI, and such loss is minimized with respect to finalizing the model for runtime usage as disclosed by DESAI)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of LUO with GOLI and DESAI as explained above. As disclosed by DESAI, one of ordinary skill would have been motivated to do so in order to “improve the reliability and speed of responses and services.” (para. 0003).
Claim 14 depends from claim 10 and claims a system that corresponds to the method of claim 7, and is therefore rejected for the same reasons explained above with respect to claims 7 and 10.
Claim 20 depends from claim 18 and claims a system that corresponds to the method of claim 7, and is therefore rejected for the same reasons explained above with respect to claims 7 and 18.
Allowable Subject Matter
Claims 8 and 15 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Claims 8 and 15 would be considered allowable because none of the references of record either alone or in combination fairly disclose or suggest the combination of limitations specified in claims 8 and 15, including at least:
apply the characterization and a ground truth for the auxiliary data to an evaluation MLM to generate a modification of the runtime settings of the MLM.
The closest prior art of record discloses:
US 20230362399 A1, hereinafter referenced as LUO, teaches a system for watermarking video content, and detecting such watermark. (paras. 0028, 0036, 0062).
US 20200356415 A1, hereinafter referenced as GOLI, teaches determining the runtime settings for an ML model as part of a runtime environment. (paras. 0013, 0028).
US 20230342677 A1, hereinafter referenced as DESAI, teaches using a loss function to measure a difference between a ground truth value and a predicted value. (para. 0146).
However, the examiner has found that the distinct feature of the Applicant's claimed invention over the prior art is the explicit claiming of the aforementioned limitations in combination with all the other limitations as specified in claims 8 and 15. Therefore, to the extent that these features are not found in the prior art cited by Examiner, claims 8 and 15 would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20240340634 A1 (Shrivastava). “Thus, if at a given time UE cannot support an AI/ML model/functionality due to limitations on its run-time capabilities, it can report it to the network and the network may configure or indicate corresponding or alternative actions to the UE such that the UE can achieve desired results and meet the KPIs from AI/ML model and device performance perspective. Triggers or events are configured by the network that may lead UE to initiate procedure to change its AI/ML model status or configuration including modification to the run-time capabilities.” (paras. 0118-0119).
US 20240171460 A1 (Hamlin). “If so, at 1004, orchestrator 501A may modify AI model parameters for any executing model at runtime according to polic(ies) 602.” (para. 0179).
US 20220327849 A1 (Albertini). “Within the present disclosure and due to the possibility to change the processing configuration and the network configuration(s) at runtime, the whole computer vision pipeline around the neural networks, with state-of-the-art algorithms, can be automatically optimized for the use-case by iterative improvement.” (para. 0057).
US 20220309314 A1 (Park). “In some embodiments, a dynamic neural network quantization logic may be configured at runtime to change the quantization, masking, and/or neural network pruning based on operating conditions, such as temperature, power consumption, utilization of processing units, etc. of an AI processor, SoC having an AI processor, memory accessed by an AI processor, and/or other peripherals of an AI processor.” (para. 0035).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL C LEE whose telephone number is (571)272-4933. The examiner can normally be reached M-F 12:00 pm - 8:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas can be reached at 571-272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL C. LEE/Examiner, Art Unit 2128
