1DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 5, 9, 12, 15-17, 20 are rejected under 35 U.S.C. 103 as being unpatentable over HARVIAINEN (US 2023/0412865 A1) in view of LEE (US 2015/0381989 A1) and LI et al. (US 2018/0167619 A1).
RE claim 1, Harviainen teaches an adaptive streaming of spatial content using multiple levels of detail (LoD) [0141-0142]. Harviainen teaches a method comprising:
(a)
receiving an original content for distribution to different client devices;
Fig. 3, content provider (308) sends spatial data (said original content) to a content server (306) [0147]. Fig. 1A, the communications system (100) may be a multiple access system that provides content to multiple wireless users [0107].
(b)
encoding the original content at different levels-of-detail (LoDs), wherein said encoding comprises generating a modified content at a first LoD that presents a creative intent from the original content with a first amount of loss;
Fig. 3, Harviainen teaches the content server (306) may compile (or generate) (312) multiple levels of detail (LoD) (said encoding the original content at different LoD) and degrees of freedom (DoF) versions of the content as content segments [0147]. The viewing client (304) may send (324) a request to the content server (306) for the lowest LoD and DoF segment for the selected viewpoint [0148]. The lowest LoD is the LoD with the smallest number of total pixels (said first LoD with a first amount of loss) [0148]. Harviainen teaches selecting LoD representations based on the quality of experience (QoE) (said a creative intent from original content), where the QoE metric may be a metric selected from the group consisting of network performance, processing performance, client computing performance, and session conditions [0015].
(c)
presenting a sample moment that corresponds to a same frame, visualization, or part of the original content and the modified content;
Harviainen teaches the process of determining the QoE metrics, selecting LoD and DoF representation based on the QoE metrics [0149]. The act of requesting LoD and DoF content segments may be an iterative process that may be continually repeated [0149]. The content server (306) may send (326) the requested segment to the viewing client (304) [0148]. The contents may be displayed (328) by the viewing client (304) and seen by the user (302) (said presenting a sample) [0148]. The user (302) may respond (330) with a user input by requesting a LoD and DoF segment according to the QoE metrics, e.g., requesting a segment with a higher LoD when QoE metrics indicate that the user experience is below a threshold [0148].
In further view of Lee, Lee teaches a method/system for incrementally improving quality and compression of media for faster transfer over data networks for improved viewing [abstract]. Server (12) executes an optimizer process (18) that optimizes such video for relatively fast retrieval and improved video quality [0021]. Fig. 2A, the ingestion process (49) includes applying (46a) a fast compression technique, frame rate reduction (46b), and conversion (46c) of the file type to one or more optimized formats for different platforms [0024]. Fig. 3, the video is available for playback [0025]. The processes (50) are iteratively applied and can be executed plural times until a stopping condition is satisfied, or the priority of additional passes becomes so low so as to obviate the need for further passes. This multi-pass process ensures that the current best version of any media is always available for viewing and downloading, while over time that same media item will be even more optimized and compressed for faster downloads and more pleasing viewing [0025]. Fig. 3A, Lee teaches determining if the current iteration result is superior to the prior best video version. Step (58) determines (76) a “video optimization score” where a higher score indicates a more desirable outcome [0033]. The process determines whether to replace (78) the video and determines (80) a priority score for a subsequent pass of the video [0033]. Lee teaches acceptable video quality scoring (72, 92) can be judged by a human and/or quality score can be generated by a computer algorithm [0035]. Human interpretation (102) can be proactive and as simple as presenting an image or video segment (said presenting a sample moment) and requesting a scaled rating from the user [0035].
It would have been obvious before the effective filing date of the claimed invention to modify Harviainen multi-LoD adaptive content-streaming system with Lee’s human-feedback-based learning technique. Harviainen teaches determining the QoE and further adjust the LoD accordingly [0148]. Lee teaches determining whether visually modified content remains acceptable based on learned human preferences [0033, 0035]. Lee further teaches this multi-pass process ensures that the current best version of any media is always available for viewing and downloading, while over time that same media item will be even more optimized and compressed for faster downloads and more pleasing viewing [0025].
(d)
receiving feedback regarding a presentation of the creative intent at the first LoD in the sample moment from the modified content;
In further view of Lee, Lee teaches acceptable video quality scoring (72, 92) can be judged by a human and/or quality score can be generated by a computer algorithm [0035]. Human interpretation (102) can be proactive and as simple as presenting an image or video segment and requesting a scaled rating (said receiving feedback) from the user [0035].
(e)
training a classification model based on the feedback, wherein training the classification model comprises determining one or more visual elements and attributes that affect the creative intent based on the feedback;
In further view of Lee, Lee teaches over a sample of videos, different techniques, and user ratings, the system can “learn” the best combination of techniques that produce the highest video optimization score (said training classification model based on the feedback) [0035]. Human rating can be used on an ongoing basis or as a set of initial training samples for a learning algorithm (1030 (e.g., a neural network) to generate a video quality score (said training classification model) [0035]. The system can evaluate (104) the best set of techniques/approaches to apply to different video categories (said determining one or more visual elements and attributes that affect the creative intent based on the feedback) [0036]. Lee teaches in videos where one or more persons are shown, the background may be irrelevant and could be blended and/or eliminated, head movements may be unimportant, while facial expressions are paramount. Videos can be manually labeled with simple categories, or the categories can be automatically detected by the system, allowing a more directed/optimized set of techniques to be applied given the contextual information, rather than a less efficient application of random or general techniques [0036].
(f)
classifying the presentation of the creative intent at the first LoD in other frames, visualizations, or parts of the modified content that are not within the sample moment using the classification model; and
In further view of Lee, Lee teaches over a sample of videos, different techniques, and user ratings, the system can “learn” the best combination of techniques that produce the highest video optimization score (said classification model) [0035]. However, Harviainen nor Lee discusses classifying the presentation of the creative intent that are not within the sample moment.
Li teaches the training subsystem (150) generates the source model (155) (said classification model) via a machine learning algorithm trained on the objective source bucket (135) and the subjective source bucket (125) [0042]. For each of the objective value sets (165), the training subsystem (150) trains the source model (155) to associate the objective value set (165) and the corresponding absolute quality score (195) [0042]. For each device type, the conversion subsystem (170(i)) generates the device equation (175(i)) that defines a relationship between the base quality score of an encode and the absolute quality score (195) for the encode viewed on a viewing device characterized by the associated additional device type [0043]. At the end of the training phase, the perceptual quality calculator (160) receives the source models (155(1)-155(3)), and the device equation (175(1)) (said classifying) [0045]. The perceptual quality calculator accurately predicts the visual quality of encodes derived from sources having different spatial resolutions as perceived when viewed on different viewing devices [0097].
It would have been obvious before the effective filing date of the claimed invention to modify Harviainen multi-LoD adaptive content-streaming system with Lee’s human-feedback-based learning technique and Li’s machine-learning perceptual-quality prediction techniques. The use of Li allows for automating the determination of the feedback of Lee. The machine-learning models trained use subjective human visual-quality scores and objective visual measurements which are also taught by Lee. Using the machine-learning models of Li and Lee allows the LoD content of Harviainen to be evaluated automatically once trained by the human evaluations are made. This would help reduce the amount of human feedback [Lee:0035].
(g)
streaming the modified content at the first LoD to a client device in response to classifying the presentation of the creative intent at the first LoD as acceptable.
Fig. 10 of Harviainen, Harviainen teaches selecting a LoD such that local rendering power is capable of rendering content segments for the selected LoD (said modified content) [0170]. As taught by Lee, Lee teaches acceptable video quality scoring (72, 92) can be judged by a human and/or quality score can be generated by a computer algorithm [0035]. Human interpretation (102) can be proactive and as simple as presenting an image or video segment and requesting a scaled rating from the user (said classifying as acceptable) [0035].
RE claim 2, Harviainen teaches further comprising:
(a)
setting a threshold at which to stream the modified content at the first LoD rather than the modified content at a second LoD; and
Harviainen teaches based on the MPD, current conditions, and current capabilities, the viewing client may select a version of the data to be downloaded (said stream first LoD rather than second LoD). Content segment format and resolution may be adapted to meet data transmission parameter and quality metric thresholds for available resources (said setting a threshold) [0141]. Furthermore, the requested (336) LoD is selected based on the QoE metrics. The QoE metrics may indicate that the user experience is below a threshold (said threshold), and thus, would request a segment higher LoD (said first LoD) [0148]. Thus, based on the capabilities and threshold, a certain version, i.e., LoD is selected for optimum viewing.
(b)
determining that one or more of a streaming performance or rendering performance associated with the client device satisfies the threshold.
Harviainen teaches the requested (336) LoD is selected based on the QoE metrics. The QoE metrics may indicate that the user experience is below a threshold (said determining performance satisfies threshold), and thus, would request a segment higher LoD (said first LoD) [0148]. Harviainen teaches the QoE metric may be a metric selected from the group consisting of network performance, processing performance, client computing performance and session conditions [0015]. Thus, the QoE metric is used to determine the quality of streaming based on the capabilities available (said determining streaming performance) [0141]. The content server streams spatial content with multiple formats and quality settings and enables a viewing client to dynamically adapt to the available bandwidth, client performance, and per session conditions [0146].
RE claim 5, Harviainen teaches wherein said encoding further comprises:
(a)
generating the modified content at a second LoD that presents the creative intent from the original content with a second amount of loss that is greater than the first amount of loss,
Fig. 3, Harviainen teaches the content server (306) may compile (or generate) (312) multiple levels of detail (LoD) (said generating modified content at a second LoD) and degrees of freedom (DoF) versions of the content as content segments [0147]. The viewing client may determine QoE metrics, i.e., network performance, processing performance, client computing performance, and session conditions [0149]. Harviainen teaches selecting LoD representations based on the quality of experience (QoE) (said a creative intent from original content), where the QoE metric may be a metric selected from the group consisting of network performance, processing performance, client computing performance, and session conditions [0015]. The LoD and DoF representations are selected based on the QoE metric [0149]. Therefore, when a LoD is selected that is below the QoE metric (said second LoD), there would be a greater amount of loss than the LoD that would be selected based on the QoE metric (said first LoD, greater amount of loss).
(b)
wherein the modified content at the second LoD is encoded with less data than the modified content at the first LoD.
Fig. 3, Harviainen teaches the content server (306) may compile (or generate) (312) multiple levels of detail (LoD) (said generating modified content at a second LoD) and degrees of freedom (DoF) versions of the content as content segments [0147]. The different LoD requires different amounts of encoded data, depending on network performance, processing performance, client computing performance, and session conditions [0015]. Thus, Harviainen teaches selecting LoD representations based on the quality of experience (QoE) (said a creative intent from original content), where the QoE metric may be a metric selected from the group consisting of network performance, processing performance, client computing performance, and session conditions [0015]. When an LoD is selected that is lower than the QoE, it will require less data than when the selected LoD for the QoE because the quality of display will be less.
RE claim 9, Harviainen teaches further comprising:
(a)
regenerating the modified content at the first LoD in response to classifying the presentation of the creative intent at the first LoD as unacceptable.
Harviainen teaches the viewing client may determine QoE metrics, such as, network performance, processing performance, client computing performance, and session conditions [0149]. The process of determining the QoE metrics, selecting LoD and DoF representations based on the QoE metric, and requesting LoD and DoF content segments may be an iterative process that may be continually repeated (said regenerating the modified content) [0149]. The LoD and DoF representations may be selected from a set of one or more LoD and DoF representations [0149]. The act of requesting LoD and DoF content segments may be an iterative process that may be continually repeated [0149]. The content server (306) may send (326) the requested segment to the viewing client (304) [0148]. The contents may be displayed (328) by the viewing client (304) and seen by the user (302) [0148]. The user (302) may respond (330) with a user input by requesting a LoD and DoF segment according to the QoE metrics, e.g., requesting a segment with a higher LoD when QoE metrics indicate that the user experience is below a threshold (said classifying the presentation of the creative intent at the first LoD as unacceptable) [0148].
RE claim 12, Harviainen in view of Lee and Li teaches wherein classifying the presentation of the creative intent comprises:
(a)
analyzing a presentation of the creative intent in each frame, visualization, or part of the other frames, visualizations, or parts against a derived model of the creative intent in the classification model; and
In further view of Li, Li teaches an absolute quality score (195) that is derived from values for the absolute perceptual quality metric assigned by humans after viewing the encodes generated by the metric engine (142(i)) on a base viewing device. The human-observed values may be obtained via any type of absolute rating process [0039]. Thus, Li teaches analyzing previously unevaluated content using a model derived from human perceptual-quality assessments.
(b)
labeling each frame, visualization, or part with a creative intent classification based on said analyzing.
Li teaches computing an absolute quality score (195) [0031-0032]. Lee further teaches an acceptable video quality scoring (72) can be judged by a human and/or a quality score can be generated by a computer algorithm. The scaled rating can be determined as “poor”, “acceptable”, “good”, or a numerical rating, e.g., 1 to 10 (said label based on analyzing) [0035].
The same motivation to combine as taught in the rationale of claim 1 is incorporated herein.
RE claim 15, claim 15 recites similar limitations as claim 1 but in system form. Therefore, the same rationale used for claim 1 is applied. Furthermore, Harviainen teaches the WTRU (102) includes a processor (118) [0121-0122].
RE claim 16, claim 16 recites similar limitations as claim 2 but in system form. Therefore, the same rationale used for claim 2 is applied.
RE claim 17, claim 17 recites similar limitations as claim 5 but in system form. Therefore, the same rationale used for claim 5 is applied.
RE claim 20, claim 20 recites similar limitations as claim 1 but in manufacture form. Therefore, the same rationale used for claim 1 is applied. Furthermore, WTRU (102) may include a non-removable memory (130) and a removable memory (132) [0121]. The processor (118) may access information from, and store data in, any type of suitable memory, such as the non-removable memory (130) and/or the removable memory (132) [0126].
Claims 3 and 4 are rejected under 35 U.S.C. 103 as being unpatentable over HARVIAINEN (US 2023/0412865 A1) in view of LEE (US 2015/0381989 A1) and LI et al. (US 2018/0167619 A1) as applied to claim 1, and in further view of RAMIREZ SOLORZANO et al. (US 2024/0177354 A1).
RE claim 3, Harviainen teaches generating different representation of 3D spatial content having different levels of detail, content formats and quality characteristics for adaptive streaming to client devices [0147-0148]. However, Harviainen in view of Lee and Li fail to discuss converting the original 3D content. Ramirez teaches a system/method of streaming 3D content [abstract]. Ramirez teaches wherein generating the modified content comprises:
(a)
converting the original content from a first three-dimensional (3D) format to a different second 3D format.
Ramirez teaches receiving data from the client (150). Received data processing unit (101), which receives the extracted instructions and/or data from the clients, further extracts the required instructions and/or data from the received data and sends them to 3D scene data creation unit (102). Then, 3D scene data creation unit (102) processes and modifies the data of 3D scene corresponding to the request managed by the server from the client in accordance with the request sent from client (150) [0036]. 3D stream conversion/encoding unit (104) converts the data received from extraction unit (103) into a 3D stream and encodes the converted data to generate 3D stream (105) [0036]. Scene reconstruction unit (155) reconstructs 3D scenes from 3D stream received from server (100) and sends the reconstructed 3D scene to display unit (156) for display to a user [0037]. The streamed 3D objects are generated from images captured at the server, and depth data of the cameras are outputted. The depth information from the camera is processed to generate a point cloud (said first 3D format). This point cloud is converted into triangles representing the actual geometry of the object and a group of triangles is generated by the server (said converting to a different second 3D format) [0067].
It would have been obvious before the effective filing date of the claimed invention to convert the input data of Harviainen as taught by Ramirez because since only arrays of vertices are transmitted as geometric data from the server to the client, the amount of data per second in the array of vertices is typically much less than that of video and frame data [Ramirez: 0073]. Additionally, since the server only sends the data of the scene that needs to be changed to the client and does not send the data of the part of the scene that is not being changed to the client, this point can also reduce the amount of data transmitted from the server to the client due to the scene change [Ramirez: 0074].
RE claim 4, Harviainen teaches generating different representations of 3D spatial content having different levels of detail, content formats and quality characteristics for adaptive streaming to client devices [0147-0148]. However, Harviainen in view of Lee and Li fail to discuss converting the original 3D content. Ramirez teaches a system/method of streaming 3D content [abstract]. Ramirez teaches wherein generating the modified content comprises:
(a)
content from one of a point, mesh, or splat 3D format to another one of the point, mesh, or splat 3D format.
Ramirez teaches receiving data from the client (150). Received data processing unit (101), which receives the extracted instructions and/or data from the clients, further extracts the required instructions and/or data from the received data and sends them to 3D scene data creation unit (102). Then, 3D scene data creation unit (102) processes and modifies the data of 3D scene corresponding to the request managed by the server from the client in accordance with the request sent from client (150) [0036]. 3D stream conversion/encoding unit (104) converts the data received from extraction unit (103) into a 3D stream and encodes the converted data to generate 3D stream (105) [0036]. Scene reconstruction unit (155) reconstructs 3D scenes from 3D stream received from server (100) and sends the reconstructed 3D scene to display unit (156) for display to a user [0037]. The streamed 3D objects are generated from images captured at the server, and depth data of the cameras are outputted. The depth information from the camera is processed to generate a point cloud (said content from one of a point). This point cloud is converted into triangles representing the actual geometry of the object and a group of triangles is generated by the server (said converting another one of mesh) [0067].
It would have been obvious before the effective filing date of the claimed invention to convert the input data of Harviainen as taught by Ramirez because since only arrays of vertices are transmitted as geometric data from the server to the client, the amount of data per second in the array of vertices is typically much less than that of video and frame data [Ramirez: 0073]. Additionally, since the server only sends the data of the scene that needs to be changed to the client and does not send the data of the part of the scene that is not being changed to the client, this point can also reduce the amount of data transmitted from the server to the client due to the scene change [Ramirez: 0074].
Claims 6 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over HARVIAINEN (US 2023/0412865 A1) in view of LEE (US 2015/0381989 A1) and LI et al. (US 2018/0167619 A1) as applied to claim 1 and 15, respectively, and in further view of BERGER (US 2008/0144727 A1).
RE claim 6, Harviainen teaches generating different representations of 3D spatial content having different levels of detail, content formats and quality characteristics for adaptive streaming to client devices [0147-0148]. However, Harviainen in view of Lee and Li fail to discuss measuring the differences between the different levels and presenting the different versions together. Berger teaches further comprising:
(a)
measuring differences between the same frame, visualization, or part of the original content and the modified content; and
Berger teaches a system/method for visually comparing (said measuring differences between the visualization) a video file against its source as the video file is being re-encoded [abstract].
wherein presenting the sample moment comprises presenting said differences in a graphical user interface with the sample moment of the original content displayed adjacent to the sample moment of the modified content.
Fig. 3, a decoded original video frame and a corresponding re-encoded video frame is displayed on the video monitor (112) side-by-side in substantially the same orientation in a wide output frame [0021].
It would have been obvious before the effective filing date of the claimed invention to provide the comparison GUI of Berger with the system method of Harviainen in view of Lee and Li because visual comparison is beneficial for detecting relatively larger differences between the images due to, for example, the transcoding process [Berger: 0021].
RE claim 18, claim 18 recites similar limitations as claim 6 but in system form. Therefore, the same rationale used for claim 6 is applied.
Claims 7 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over HARVIAINEN (US 2023/0412865 A1) in view of LEE (US 2015/0381989 A1) and LI et al. (US 2018/0167619 A1) as applied to claim 1, and in further view of vanZWOL et al. (US 2013/0142418 A1).
RE claim 7, Harviainen in view of Lee and Li teaches the limitations of claim 7 with the exception of selecting a sample moment. vanZwol teaches selecting representative images for video items [abstract, 0020]. vanZwol teaches further comprising:
(a)
detecting a first scene and a different second scene in the original content; and
vanZwol teaches a scene selection mechanism. Specifically, the boundaries between scenes within a video are identified and one or more frames are selected from which each scene [0028].
wherein selecting the sample moment comprises:
(b)
extracting a first frame, visualization, or part representing the creative intent in the first scene and a second frame, visualization, or part representing the creative intent in the different second scene from the original content and the modified content.
Once scene boundaries are detected, one or more candidate images are selected from each scene (said extracting a first/second part representing the first/second scene) [0033] Once the candidate images have been extracted from the video, a trained machine learning engine is used to determine which image are most representative of the video [0035]. Fig. 3, automatic selection of a representative image is performed during the scoring phase by feeding parameter values (306) associated with each candidate image of a video into the machine learning engine (308) [0036]. The candidate image with the highest score may be selected as the representative image [0115]. Furthermore, as combined with Harviainen in view of Lee and Li, the different LoD (said original and modified content) of Harviainen correspond portions of the same underlying content. It would have been obvious before the effective filing date of the claimed invention to extract the frames at corresponding temporal positions from both representations to facilitate consistent comparison of perceptually image visual characteristics (said original content and the modified content).
It would have been obvious before the effective filing date of the claimed invention to extract multiple representative images of the video of Harviainen in view of Lee and Li. vanZwol teaches it may be desirable to select multiple representative images for a single video to provide a “story” told by the video item [0117]. This would provide a quick summary of the video content.
RE claim 8, in further view of vanZwol, vanZwol teaches wherein training the classification model comprises:
(a)
training a first classification model for detecting and classifying the creative intent in the first scene based on feedback received for the first frame, visualization, or part; and
Fig. 3, vanZwol depicts the training and scoring phases of a machine learning engine (308). During the training phase, the machine learning engine (308) goes through a series of training operations. During each training operation, the machine learning engine (308) is fed an externally generated score, and a set of input parameter values associated with the score [0023]. To create the training set, candidate images are assigned scores, which may be humans manually assigning scores (said based on feedback) [0052]. The scores may be used to train the machine learning engine (308), and the trained machine learning engine (308) may then be used to generate representativeness scores for the remainder of the video collection. As taught in the rationale of claim 7, vanZwol further teaches determining boundaries between scenes [0028]. Therefore, different scenes can be trained accordingly (said first scene).
(b)
training a second classification model for detecting and classifying the creative intent in the different second scene based on feedback received for the second frame, visualization, or part.
Fig. 3, vanZwol depicts the training and scoring phases of a machine learning engine (308). During the training phase, the machine learning engine (308) goes through a series of training operations. During each training operation, the machine learning engine (308) is fed an externally generated score, and a set of input parameter values associated with the score [0023]. To create the training set, candidate images are assigned scores, which may be humans manually assigning scores (said based on feedback) [0052]. The scores may be used to train the machine learning engine (308), and the trained machine learning engine (308) may then be used to generate representativeness scores for the remainder of the video collection. As taught in the rationale of claim 7, vanZwol further teaches determining boundaries between scenes [0028]. Therefore, different scenes can be trained accordingly (said second scene).
The same motivation to combine as taught in the rationale of claim 7 is incorporated herein.
Claims 10 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over HARVIAINEN (US 2023/0412865 A1) in view of LEE (US 2015/0381989 A1) and LI et al. (US 2018/0167619 A1) as applied to claim 1 and 15 respectively, and in further view of BESENBRUCH et al. (2023/0388501 A1).
RE claim 10, Harviainen in view of Lee and Li teaches the limitations of claim 10 with the exception of teaching the art of adjusting a plurality of loss functions of a generative model in response to determining that the creative intent is lost. Besenbruch teaches training neural networks for loss image or video compressing using loss functions that control the amount of distortion introduced into different image segments. Harviainen in view of Lee, Li, and Besenbruch teaches further comprising:
(a)
determining that the creative intent as presented in modified content at the first LoD is lost based on said classifying; and
In view of Lee, Lee teaches as each iteration through the video optimization process completes, analysis (58) is performed to evaluate the iteration results [0031]. The analysis (48) determines (76) a “video optimization score” (said classifying) where a high score indicates a more desirable outcome [0033]. Acceptable video quality scoring (72) can be judged by a human and/or a quality score can be generated by a computer algorithm [0035].
In further view of Li, Li further teaches a perceptual quality calculator to accurately predict the visual quality of encodes derived from sources having different spatial resolutions as perceived when viewing on different viewing devices [0097]. The absolute quality scores allow meaningful comparison and effective optimization operations across encodes derived from a wide range of sources [0097].
Therefore, Lee and Li teach determining whether modified content retains acceptable perceptual quality using a trained model. Based on the scores, it is determined if the selected LoD of Harviainen provides the most desirable output (said determining that the creative intent … is lost).
In further view of Besenbruch, Besenbruch teaches using deep learning and AI to provide an improved compression and decompression schemes [0821]. Besenbruch teaches utilizing a number of different learnt compression pipelines with distinctly different distortion metrics applied on the same segmentations of the images [0894]. Once trained, human raters are asked in a 2AFC selection procedure to indicate which patch from the trained compression pipelines produces the perceptually most pleasing image patch (said determining that the creative intent … is lost). [0894]. Thus, a large dataset with image patches and labels is then formed, where the label indicates which distortion metric is preferred. A neural network classifier is then trained to predict the optimal distortion metric for each patch [0894].
(b)
adjusting a plurality of loss functions of a generative model in response to determining that the creative intent is lost,
In view of Besenbruch, Besenbruch teaches applying an additional transformation to the segmented data [0886-0887]. A weighted mixture of multiple existing loss functions can be used to predict the visual loss score [1143].
wherein adjusting the plurality of loss functions comprises:
(c)
configuring a first loss function of the plurality of loss functions that controls a first visual element or attribute that affects the creative intent with a first amount of loss; and
Besenbruch teaches the computation Ri means that each segment can have a variable rate. For example, assigning more bits to regions with higher sensitivity for the HVS, such as the faces and texts, or any other salient region in the image (said first visual element), will improve perceptual quality without increasing the total number of bits required for the compressed media [0888].
(d)
configuring a second loss function of the plurality of loss functions that controls a second visual element or attribute that does not affect the creative intent with a second amount of loss, wherein the second amount of loss is greater than the first amount of loss.
Besenbruch teaches HVSA is more sensitive to changes in low frequency regions, such as uniform areas, compared to changes in high frequency regions such as patches of grass. In general, for most images the majority of high frequencies (said second visual element) can be removed within any noticeable difference in the image [0892].
It would have been obvious before the effective filing date of the claimed invention to modify the combined invention of Harviainen in view of Lee and Li with the neural-network compression and loss-function configuration techniques of Besenbruch. One would be motivated to use the perceptual-quality classification results of Lee and Li to identify when an encoded LoD representation preserves important visual characteristics, and in response, adjust the distortion objects of the neural-network-based compression model to prioritize preservation of those characteristics. This modification would predictably improve the viewing quality of the encoded content while permitting greater compression of less important visual information, thereby reducing bandwidth requirements without having to degrade the user’s viewing experience.
RE claim 19, claim 19 recites similar limitations as claim 10 but in system form. Therefore, the same rationale used for claim 10 is applied.
Claims 13 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over HARVIAINEN (US 2023/0412865 A1) in view of LEE (US 2015/0381989 A1) and LI et al. (US 2018/0167619 A1) as applied to claim 1, and in further view of WU et al. (US 2017/0256065 A1).
RE claim 13, Harviainen in view of Lee and Li teaches the limitations of claim 13 with the exception of determining when the particular part changes by more than a threshold amount. Wu is made of record as teaching the art of tracking regions of interest across video frames. Wu teaches wherein training the classification model further comprises:
(a)
isolating the creative intent to a particular part of the sample moment; and
In view of Lee, Lee teaches the system can evaluate (104) the best set of techniques/approaches to apply to different video categories such as where people are shown, the background may be irrelevant and could be blended and/or eliminated, while facial expressions are paramount [0036].
In view of Wu, Wu teaches tracking regions of interest across video frames [abstract]. A region of interest (ROI) may be indicated by a user and may be associated with or represent a region, object, or the like that a user or the device indicates for tracking [0021, 0030]. The user may click on the person or object, and the ROI tracking techniques may attempt to track the clicked person or object across video frames [0021]. Fig. 1, system (100) includes a tracking module (102) that may receive a ROI signal (132) and ROI center (161) [0027-0028]. A region generation module (111) may generate ROI (141) around the pixel or pixel location indicated by ROI signal (132) (said isolating a particular part) [0032].
wherein classifying the presentation of the creative intent comprises:
(b)
determining when the particular part representing the creative intent changes by more than a threshold amount in the other frames, visualizations, or parts of the modified content.
In further view of Wu, Wu teaches tracking a region of interest across video frames [0063, Fig. 4]. Video frame (503), feature points (602), and effective center (603) may be received (401) [0065-0066]. Motion tracking of feature points may be performed from the previous frame to the current frame [0068]. The color and/or depth differences may be compared to a threshold and, if both are less than (e.g., do not exceed) their respective thresholds, the feature point for the current frame may be accepted and, if either exceeds its respective threshold, the feature point for the current frame may be rejected [0069]. Therefore, in the combined invention, the ROI of Wu corresponds to the visually important regions of Harviainen in view of Lee and Li.
It would have been obvious before the effective filing date of the claimed invention to modify the combined system of Harviainen in view of Lee and Li to incorporate the ROI tracking of Wu. Wu teaches the tracking techniques that may allow for compelling 3D video effects such as video refocusing, video segmentation, video layer effects, or the like [Wu: 0020]. This would support the teachings of Lee in regard to determining the different parts of the video and adapting different techniques according to their classification [Lee: 0036].
RE claim 14, in further view of Wu, Wu teaches wherein classifying the presentation of the creative intent further comprises:
(a)
labeling a frame, visualization, or part of the other frames, visualizations, or parts of the modified content as losing the creative intent in response to determining that the particular part in that frame, visualization, or part changes by more than the threshold amount.
As taught in the rationale of claim 13(b), Wu teaches the features are accepted based on thresholds [0069]. Wu teaches when both differences do not exceed their respective thresholds, the feature point may be kept and if either difference exceeds its respective threshold, the feature point may be discarded [0023].
Li teaches computing an absolute quality score (195) [0031-0032]. Lee further teaches an acceptable video quality scoring (72) can be judged by a human and/or a quality score can be generated by a computer algorithm. The scaled rating can be determined as “poor”, “acceptable”, “good”, or a numerical rating, e.g., 1 to 10 (said classifying) [0035].
Lee additionally teaches acceptable video quality scoring (72, 92) can be judged by a human and/or quality score can be generated by a computer algorithm [0035]. Human interpretation (102) can be proactive and as simple as presenting an image or video segment and requesting a scaled rating (said classifying) from the user [0035].
The same motivation to combine as taught in the rationale of claim 13 is incorporated herein.
Allowable Subject Matter
Claim 11 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is an examiner’s statement of reasons for allowance:
Reference HARVIAINEN (US 2023/0412865 A1) is made of record as teaching an adaptive streaming of spatial content using multiple levels of detail (LoD) [0141-0142]. Fig. 3, Harviainen teaches the content server (306) may compile (or generate) (312) multiple levels of detail (LoD) and degrees of freedom (DoF) versions of the content as content segments [0147]. The viewing client (304) may send (324) a request to the content server (306) for the lowest LoD and DoF segment for the selected viewpoint [0148]. The lowest LoD is the LoD with the smallest number of total pixels [0148]. Harviainen teaches selecting LoD representations based on the quality of experience (QoE), where the QoE metric may be a metric selected from the group consisting of network performance, processing performance, client computing performance, and session conditions [0015].
Reference LEE (US 2015/0381989 A1) is made of record as teaching a method/system for incrementally improving quality and compression of media for faster transfer over data networks for improved viewing [abstract]. Server (12) executes an optimizer process (18) that optimizes such video for relatively fast retrieval and improved video quality [0021]. Fig. 2A, the ingestion process (49) includes applying (46a) a fast compression technique, frame rate reduction (46b), and conversion (46c) of the file type to one or more optimized formats for different platforms [0024]. Fig. 3, the video is available for playback [0025]. The processes (50) are iteratively applied and can be executed plural times until a stopping condition is satisfied, or the priority of additional passes becomes so low so as to obviate the need for further passes. This multi-pass process ensures that the current best version of any media is always available for viewing and downloading, while over time that same media item will be even more optimized and compressed for faster downloads and more pleasing viewing [0025]. Fig. 3A, Lee teaches determining if the current iteration result is superior to the prior best video version. Step (58) determines (76) a “video optimization score” where a higher score indicates a more desirable outcome [0033]. The process determines whether to replace (78) the video and determines (80) a priority score for a subsequent pass of the video [0033]. Lee teaches acceptable video quality scoring (72, 92) can be judged by a human and/or quality score can be generated by a computer algorithm [0035]. Human interpretation (102) can be proactive and as simple as presenting an image or video segment and requesting a scaled rating from the user [0035].
Reference LI et al. (US 2018/0167619 A1) is made of record as teaching a training subsystem (150) generates the source model (155) via a machine learning algorithm trained on the objective source bucket (135) and the subjective source bucket (125) [0042]. For each of the objective value sets (165), the training subsystem (150) trains the source model (155) to associate the objective value set (165) and the corresponding absolute quality score (195) [0042]. For each device type, the conversion subsystem (170(i)) generates the device equation (175(i)) that defines a relationship between the base quality score of an encode and the absolute quality score (195) for the encode viewed on a viewing device characterized by the associated additional device type [0043]. At the end of the training phase, the perceptual quality calculator (160) receives the source models (155(1)-155(3)), and the device equation (175(1)) [0045]. The perceptual quality calculator accurately predicts the visual quality of encodes derived from sources having different spatial resolutions as perceived when viewed on different viewing devices [0097].
Reference BESENBRUCH et al. (2023/0388501 A1) is made of record as teaching the art of using deep learning and AI to provide an improved compression and decompression schemes [0821]. Besenbruch teaches utilizing a number of different learnt compression pipelines with distinctly different distortion metrics applied on the same segmentations of the images [0894]. Once trained, human raters are asked in a 2AFC selection procedure to indicate which patch from the trained compression pipelines produces the perceptually most pleasing image patch. [0894]. Thus, a large dataset with image patches and labels is then formed, where the label indicates which distortion metric is preferred. A neural network classifier is then trained to predict the optimal distortion metric for each patch [0894].
However, the cited prior art does not disclose or render obvious the combination of elements recited in the claims as whole. Specifically, the cited prior art fails to disclose or render obvious the limitations: regenerating the modified content at the first LoD with a new set of primitives that are defined with visual elements and attributes that match the first amount of loss of the configured first loss function and the second amount of loss of the configured second loss function.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE L SAMS:
direct telephone number:
(571) 272-7661
email:
michelle.sams@uspto.gov
personal fax number:
(571)273-7661
The examiner is currently part-time and can be reached Mon.-Fri. 5:30am-9:30am.
Examiner interviews are available via telephone and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee M. Tung can be reached on (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE L SAMS/
Primary Examiner, Art Unit 2611
15 October 2026