Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Priority
Acknowledgment is made of applicant’s foreign priority claim, for U.S. Application No. 18/724,491, based on a foreign application filed on 01/24/2022.
Status of Claims
Claims 1–11, 13-14, 16-22 are pending in the application. Claims 1–11, 13-14, 16-22 are rejected. Claims 12, 15 are canceled.
Overview of Grounds of Rejection
Ground of Rejection
Claim(s)
Statute(s)
Reference(s)
Ground of Rejection 1
1
§ 103
Kim et al. (US20210006755A1) in view of Chang et al. (US20190251674A1), and further in view of White et al. (US20200104970A1)
Ground of Rejection 2
2-4, 10
§ 103
Kim et al. (US20210006755A1) in view of Lugaresi et al. (NPL)
Ground of Rejection 3
5-6, 8
§ 103
Kim et al. (US20210006755A1) in view of White et al. (US20200104970A1)
Ground of Rejection 4
7
§ 103
Kim et al. (US20210006755A1) in view of Chang et al. (US20190251674A1), and further in view of Lugaresi et al. (NPL)
Ground of Rejection 5
9
§ 103
Kim et al. (US20210006755A1) in view of Rymkowski et al. (US20180082407A1)
Ground of Rejection 6
11
§ 103
Kim et al. (US20210006755A1) in view of Chang et al. (US20190251674A1)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
(Please see the cited paragraphs, sections, pages, or surrounding text in the references for the paraphrased content.)
Ground of Rejection 1
Claims 1, 13, 14 are rejected under 35 U.S.C. § 103 as being unpatentable over Kim et al. (US20210006755A1) in view of Chang et al. (US20190251674A1), and further in view of White et al. (US20200104970A1).
As per Claim 1, Kim teaches the following portion of Claim 1, which recites:
“An image processing method, comprising:”
Kim et al. teaches an image-processing method. Kim et al. states: “there is provided an operating method of an image processing device” and further teaches “select[ing] a processing mode,” “perform[ing] image processing on the image data by using a neural network processor and processing circuitry,” and alternatively processing “apart from the neural network processor.” This teaches the claimed image processing method. — Kim et al., ¶¶ [0007], [0060].
Kim alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Chang and White, they collectively teach all of the limitation(s).
Kim, Chang and White teach the following portion of Claim 1, which recites:
“obtaining configuration information matching effect editing in response to performing the effect editing on an initial image, wherein the configuration information comprises a deep learning inference node for performing the effect editing on the initial image, a pre-processing function node associated with the deep learning inference node and a post-processing function node associated with the deep learning inference node;”
Kim et al. teaches obtaining image data and selecting a processing mode, but Kim et al. alone does not clearly teach the claimed configuration information matching effect editing with the claimed node terminology. Kim et al. states that “the image processing device may be configured to select a processing mode” and that the system includes a “pre-processor,” “neural network processor,” and “main processor.” — Kim et al., ¶¶ [0044], [0047], [0060]-[0062].
Chang et al. teaches the claimed effect editing on an initial image. Chang et al. states that “automatic retouching tool 100 performs digital image processing of a specified input such as a designated photograph,” and further that “automatic retouching tool 100 is tailored to automatically retouch” the image. Chang et al. also teaches that “the high and low frequency layers are processed using high frequency path 125 and low frequency path 155” and that “each of high and low frequency paths 125 and 155 advantageously utilizes a supervised deep learning technique.” This teaches effect editing of an initial image using deep learning. — Chang et al., ¶¶ [0029], [0031].
White et al. teaches the claimed configuration information and node structure. White et al. states that a “render graph asset” is “represented as a JavaScript Object Notation (JSON) data file,” that the system creates “data file[s]” for render graphs, and that “The render graph asset is able to define a collection of nodes, render targets, and the connections between nodes and render targets.” White et al. further teaches that “Each of the nodes consist of a setup function and an execute function” and that the render graph file “can define the inputs, outputs and settings for the nodes in a render graph.” This teaches configuration information comprising node definitions and associated functions. — White et al., ¶¶ [0017], [0031]-[0032], [0035]-[0037].
Kim and Chang teach the following portion of Claim 1, which recites:
“calling processing logic of the pre-processing function node based on the configuration information to transform the initial image by the processing logic of the pre-processing function node to obtain input data meeting an input requirement of an algorithm model corresponding to the deep learning inference node;”
Kim et al. teaches pre-processing generally, but Chang et al. more closely teaches this limitation in the context of effect editing. Chang et al. states that “Skin filter 110 filters out all non-skin regions to produce a skin mask and a residual image,” that “the skin mask is used as an input into skin quality detection network 120 and frequency separator 115,” and that “the skin quality map is fed into both high and low frequency paths 125 and 155 as an input.” Chang et al. further teaches that “the skin mask is fed into frequency separator 115, which separates the skin mask into a high frequency layer and a low frequency layer.” This teaches transforming the initial image through pre-processing to obtain input data suitable for the deep-learning model. — Chang et al., ¶¶ [0030]-[0031].
Chang teaches the following portion of Claim 1, which recites:
“performing the effect editing on the initial image based on the input data by the algorithm model corresponding to the deep learning inference node to obtain output data;”
Chang et al. teaches this limitation. Chang et al. states that “high frequency layer 125 utilizes a conditional generative adversarial network to automatically retouch the high frequency layer,” and that “Low frequency layer 155 utilizes a neural network trained to generate a bilateral grid ... [and] the upsampled color transformation is applied to automatically retouch the low frequency layer.” Chang et al. also teaches, in method form, that “At block 330, the high frequency layer is automatically retouched, using a first neural network that accepts the skin quality map as an input,” and “At block 340, the low frequency layer is automatically retouched ... using a second neural network and the skin quality map.” This teaches performing the claimed effect editing by the algorithm model to obtain output data. — Chang et al., ¶¶ [0031], [0056].
Chang teaches the following portion of Claim 1, which recites:
“calling processing logic of the post-processing function node based on the configuration information to transform the output data by the processing logic of the post-processing function node to obtain a target image added with an effect.”
Chang et al. teaches this limitation. Chang et al. states that “The automatically retouched high and low frequency layers are fed into combiner 175 ... to reconstitute the high and low frequency layers and the residual image into a retouched photo.” Chang et al. also teaches that “At block 350, the retouched high frequency layer and the retouched low frequency layer are combined to generate a combined retouched image.” This teaches post-processing the output data to obtain a final target image having the applied effect, namely a retouched photo. — Chang et al., ¶¶ [0031], [0056].
Before the effective filing date of the claimed invention, a person of ordinary skill in the art would have been motivated to combine Kim et al., Chang et al., and White et al. because Kim et al. teaches a modular image-processing pipeline using pre-processing, a neural network processor, and post-processing; Chang et al. teaches applying deep-learning-based automatic retouching to an input image; and White et al. teaches implementing processing pipelines through reusable configuration data that defines nodes, inputs, outputs, and associated functions. A POSITA would have found it obvious to organize Chang et al.’s deep-learning image effect editing within Kim et al.’s image-processing architecture and to express that pipeline using White et al.’s node-based configuration approach, yielding a more configurable and modular system with predictable improvements in processing control, reuse, and deployment.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Claim 13 does not include any additional limitations that would significantly distinguish it from claim 1. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Claim 14 does not include any additional limitations that would significantly distinguish it from claim 1. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Ground of Rejection 2
Claims 2, 3, 4, 10, 16, 17, 18, 20, 21, 22 are rejected under 35 U.S.C. § 103 as being unpatentable over Kim et al. (US20210006755A1) in view of Lugaresi et al. (NPL).
As per Claim 2, Kim alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Lugaresi, they collectively teach all of the limitation(s).
Kim and Lugaresi teach Claim 2, which recites:
“The image processing method according to claim 1, wherein: pre-processing function nodes associated with different deep learning inference nodes are the same; and/or post-processing functional nodes associated with different deep learning inference nodes are the same.”
Kim et al. teaches the base image processing method according to claim 1, including an image-processing pipeline having a “pre-processor 100,” “neural network processor 200,” and “main processor 300,” where the system performs image processing using the neural network processor and other processing circuitry. However, Kim et al. does not clearly teach that the same pre-processing function node and/or the same post-processing function node is associated with different deep learning inference nodes. — Kim et al., ¶¶ [0005]-[0007], [0044], [0047], [0060].
Lugaresi et al. teaches this missing limitation. Lugaresi et al. states that “a perception pipeline can be built as a graph of modular components, including model inference, media processing algorithms and data transformations” and that “Each node in the graph is implemented as a Calculator.” Lugaresi et al. further teaches, in the face-landmark-detection-and-segmentation pipeline, “a demultiplexing node that splits the packets in the input stream into interleaving subsets of packets, with each subset going into a separate output stream.” This teaches the same pre-processing function node associated with different deep learning inference nodes, namely the FaceLandmarkDetection node and the FaceSegmentation node shown in Figure 5. Lugaresi et al. also teaches “for visualization the annotations from the two tasks are overlaid onto the camera frames, with synchronization across the three streams handled by the input policy of the annotation node.” This teaches the same post-processing functional node associated with different deep learning inference nodes, because the single annotation node post-processes outputs from both inference branches. — Lugaresi et al. (NPL), pp. 1, 3, 8, Fig. 5.
Before the effective filing date of the claimed invention, a person of ordinary skill in the art would have been motivated to combine Kim et al. with Lugaresi et al. because Kim et al. teaches an image-processing architecture using a pre-processor, a neural network processor, and post-processing circuitry, while Lugaresi et al. teaches implementing inference pipelines as a reusable graph of modular components in which the same upstream or downstream node can serve multiple inference branches. A POSITA would have found it obvious to organize Kim et al.’s image-processing pipeline so that common pre-processing and/or post-processing nodes are shared across multiple deep learning inference nodes, as taught by Lugaresi et al., in order to improve modularity, reduce duplicated processing, and obtain predictable implementation and resource-efficiency benefits.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
As per Claim 3, Kim alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Lugaresi, they collectively teach all of the limitation(s).
Kim and Lugaresi teach Claim 3, which recites:
“The image processing method according to claim 1, wherein the pre-processing function node comprises a plurality of nodes, and the configuration information further comprises an execution sequence of the plurality of nodes.”
Kim et al. teaches the base image-processing method of claim 1, including a pipeline having a “pre-processor 100,” “neural network processor 200,” and “main processor 300,” and further teaches that the pre-processor may include multiple pre-processing modules, e.g., “an X-talk correction module 120 a and a despeckle module 130 a.” However, Kim et al. does not clearly teach that the pre-processing function node itself comprises a plurality of nodes, nor that the configuration information includes an execution sequence of those nodes. — Kim et al., ¶¶ [0044], [0047], [0061]-[0063].
Lugaresi et al. teaches the missing limitation. Lugaresi et al. states that “A pipeline is defined as a directed graph of components where each component is a Calculator,” that “A GraphConfig is a specification that describes the topology and functionality of a MediaPipe graph,” and that “a MediaPipe graph can be defined as a Subgraph,” where “each subgraph node is replaced by the corresponding graph of calculators.” Thus, Lugaresi et al. teaches that a single functional node may comprise a plurality of nodes. Lugaresi et al. further teaches that the graph is defined by connected calculators and streams, and that the graph configuration specifies node type, inputs, and outputs, thereby teaching configuration information that includes the ordered graph structure, i.e., the execution sequence of the plurality of nodes. — Lugaresi et al. (NPL), Sections 3, 3.5, 3.6, pp. 2-4.
Before the effective filing date of the claimed invention, a person of ordinary skill in the art would have been motivated to combine Kim et al. with Lugaresi et al. because Kim et al. teaches an image-processing architecture with pre-processing, inference, and post-processing components, while Lugaresi et al. teaches representing such processing as a configurable graph in which a functional block may be implemented as a subgraph containing multiple nodes with a defined topology and execution flow. A POSITA would have found it obvious to implement Kim et al.’s pre-processing function as a plurality of graph nodes whose sequence is specified in configuration information, as taught by Lugaresi et al., in order to improve modularity, reuse, maintainability, and predictable pipeline control.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
As per Claim 4, Kim alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Lugaresi, they collectively teach all of the limitation(s).
Kim and Lugaresi teach Claim 4, which recites:
“The image processing method according to claim 3, wherein the calling of the processing logic of the pre-processing function node based on the configuration information comprises: sequentially calling processing logic of the plurality nodes in the pre-processing function node according to the execution sequence.”
Kim et al. teaches the base image-processing method and a multi-stage pre-processing arrangement. Kim et al. states that the “pre-processor 100 a may include an X-talk correction module 120 a and a despeckle module 130 a,” and that the pre-processor performs “pre-processing including an X-talk correction and/or despeckle operation.” However, Kim et al. does not clearly teach the claimed feature of sequentially calling the processing logic of a plurality of pre-processing nodes according to the execution sequence defined by configuration information. — Kim et al., ¶¶ [0062]-[0063].
Lugaresi et al. teaches the missing limitation. Lugaresi et al. states that “A pipeline is defined as a directed graph of components where each component is a Calculator,” that “A GraphConfig is a specification that describes the topology and functionality of a MediaPipe graph,” and that “each subgraph node is replaced by the corresponding graph of calculators.” Lugaresi et al. further teaches that calculators execute through “Open(), Process() and Close(),” that “When the graph is initialized, nodes are topologically sorted and assigned a priority based on the graph’s layout,” and that “Input sets are processed in strictly ascending timestamp order.” These teachings show sequentially calling processing logic of the plurality nodes according to an execution sequence defined by the graph topology / configuration. — Lugaresi et al. (NPL), Sections 3, 3.4, 3.6, 4.1.1, 4.1.3, pp. 2-5.
Before the effective filing date of the claimed invention, a person of ordinary skill in the art would have been motivated to combine Kim et al. with Lugaresi et al. because Kim et al. teaches a pre-processing pipeline with multiple pre-processing modules, while Lugaresi et al. teaches implementing such multi-stage processing as graph nodes whose execution order is defined by configuration and graph topology. A POSITA would have found it obvious to arrange Kim et al.’s plurality of pre-processing modules as configurable nodes that are called according to an execution sequence, as taught by Lugaresi et al., in order to improve modularity, configurability, maintainability, and predictable execution control.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
As per Claim 10, Kim alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Lugaresi, they collectively teach all of the limitation(s).
Kim and Lugaresi teach Claim 10, which recites:
“The image processing method according to claim 1, wherein the algorithm model corresponding to the deep learning inference node, the processing logic of the pre-processing function node, and the processing logic of the post-processing function node are stored separately from the configuration information.”
Kim et al. teaches the base image-processing method of claim 1, including separate processing components such as a “pre-processor 100,” “neural network processor 200,” and “main processor 300,” and teaches corresponding pre-processing, inference, and post-processing operations. However, Kim et al. does not clearly teach that the algorithm model and the processing logic for the pre-processing and post-processing function nodes are stored separately from the configuration information. — Kim et al., ¶¶ [0044], [0047], [0061]-[0066].
Lugaresi et al. teaches the missing limitation. Lugaresi et al. states that “A GraphConfig is a specification that describes the topology and functionality of a MediaPipe graph,” and that “Each calculator included in a program is registered with the framework so that the graph configuration ... can reference it by name.” This teaches that the processing logic of the nodes is separate from the configuration information. Lugaresi et al. also teaches that “the string defining the file path for an ML model can be fed into a node through a side packet,” and that “The object-detection node consumes an ML model and the associated label map as input side packets.” This teaches that the algorithm model is likewise stored separately from the configuration information. — Lugaresi et al. (NPL), Sections 3, 3.3, 3.4, 3.6, 6.1, pp. 2-4, 7.
Before the effective filing date of the claimed invention, a person of ordinary skill in the art would have been motivated to combine Kim et al. with Lugaresi et al. because Kim et al. teaches an image-processing architecture having separate pre-processing, inference, and post-processing components, while Lugaresi et al. teaches a graph-based configuration in which configuration information is separated from both node-processing logic and model assets. A POSITA would have found it obvious to store Kim et al.’s algorithm model and node logic separately from configuration information, as taught by Lugaresi et al., in order to improve modularity, maintainability, configurability, and predictable reuse of models and processing components across different pipelines.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Claim 16 does not include any additional limitations that would significantly distinguish it from claim 2. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Claim 17 does not include any additional limitations that would significantly distinguish it from claim 3. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Claim 18 does not include any additional limitations that would significantly distinguish it from claim 4. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Claim 20 does not include any additional limitations that would significantly distinguish it from claim 2. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Claim 21 does not include any additional limitations that would significantly distinguish it from claim 3. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Claim 22 does not include any additional limitations that would significantly distinguish it from claim 4. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Ground of Rejection 3
Claims 5, 6, 8, 19 are rejected under 35 U.S.C. § 103 as being unpatentable over Kim et al. (US20210006755A1) in view of White et al. (US20200104970A1).
As per Claim 5, Kim alone does not explicitly teach all the limitation(s) of the claim. However, when combined with White, they collectively teach all of the limitation(s).
Kim and White teach Claim 5, which recites:
“The image processing method according to claim 1, wherein the post-processing function node comprises a plurality of nodes, and the configuration information further comprises an execution sequence of the plurality of nodes.”
Kim et al. teaches the base image-processing method of claim 1 and teaches post-processing generally. Kim et al. states that the main processor may perform “post-processing including a Bayer demosaic operation, a Bayer denoising operation, and/or a sharpening operation,” and also teaches that a configuration may include “a greater number of ... post-processing operations.” However, Kim et al. does not clearly teach that a post-processing function node itself comprises a plurality of nodes, or that configuration information includes an execution sequence of those nodes. — Kim et al., ¶¶ [0061]-[0063].
White et al. teaches the missing limitation. White et al. states that “the post process render graph 402C represents a rendering pipeline for a post process pass,” and that “the post process render graph 402C includes a bloom downsample node 404G, luminance calculation node 404H, and a post process combined node 404I.” This teaches that the post-processing function node comprises a plurality of nodes. White et al. further teaches that “the render graph file 508 is a JSON file that defines the collection of nodes ... and the connections between the nodes,” and that the render graphs are “sequentially connected together.” This teaches configuration information comprising the execution sequence of the plurality of nodes. — White et al., ¶¶ [0028]-[0029], [0031]-[0032], [0035]-[0036].
Before the effective filing date of the claimed invention, a person of ordinary skill in the art would have been motivated to combine Kim et al. with White et al. because Kim et al. teaches an image-processing architecture with post-processing operations following neural-network-based processing, while White et al. teaches implementing post-processing as a configurable graph made of multiple nodes whose connections define execution order. A POSITA would have found it obvious to implement Kim et al.’s post-processing functionality as multiple configured nodes arranged in sequence, as taught by White et al., in order to improve modularity, configurability, reuse, and predictable pipeline control.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
As per Claim 6, Kim alone does not explicitly teach all the limitation(s) of the claim. However, when combined with White, they collectively teach all of the limitation(s).
Kim and White teach Claim 6, which recites:
“The image processing method according to claim 5, wherein the calling of the processing logic of the post-processing function node based on the configuration information comprises: sequentially calling processing logic of the plurality of nodes in the post-processing function node according to the execution sequence.”
Kim et al. teaches the base image-processing method and post-processing generally. Kim et al. states that the “main processor 300 a may be configured to receive the Bayer data IDATAb and/or to generate RGB data IDATAc by performing post-processing including a Bayer demosaic operation, a Bayer denoising operation, and/or a sharpening operation” and further teaches a configuration that may include “a greater number of ... post-processing operations.” However, Kim et al. does not clearly teach sequentially calling the logic of a plurality of nodes in a post-processing function node according to the execution sequence. — Kim et al., ¶¶ [0061]-[0063].
White et al. teaches the missing limitation. White et al. states that “the render frame 400 includes three different render graphs 402A-402C sequentially connected together,” that “the post process render graph 402C represents a rendering pipeline for a post process pass,” and that “the post process render graph 402C includes a bloom downsample node 404G, luminance calculation node 404H, and a post process combined node 404I.” White et al. further teaches that “the render graph file 508 is a JSON file that defines the collection of nodes ... and the connections between the nodes and render targets,” and that “each of the nodes consist of a setup function and an execute function.” These teachings show sequentially calling processing logic of the plurality of nodes in the post-processing function node according to configured execution order. — White et al., ¶¶ [0028]-[0032], [0035]-[0036].
Before the effective filing date of the claimed invention, a person of ordinary skill in the art would have been motivated to combine Kim et al. with White et al. because Kim et al. teaches an image-processing architecture having post-processing after neural-network-based processing, while White et al. teaches implementing post-processing as a configured render graph made of multiple nodes connected in sequence and executed according to the graph definition. A POSITA would have found it obvious to implement Kim et al.’s post-processing functionality as sequentially called configured nodes, as taught by White et al., in order to improve modularity, configurability, reuse, and predictable execution control.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
As per Claim 8, Kim alone does not explicitly teach all the limitation(s) of the claim. However, when combined with White, they collectively teach all of the limitation(s).
Kim and White teach Claim 8, which recites:
“The image processing method according to claim 1, wherein the obtaining of the configuration information matching the effect editing comprises: determining a configuration file having a preset binding relationship with the effect editing; and reading the configuration information from the configuration file.”
Kim et al. teaches the base image-processing method of claim 1, including selecting a processing mode and generating second and third image data using a neural network processor and post-processing circuitry. Kim et al. also teaches that the neural network processor and processing circuitry may operate using stored data and configuration-related program material in memory / storage. However, Kim et al. does not clearly teach determining a configuration file bound to the selected editing operation and then reading configuration information from that file. — Kim et al., ¶¶ [0005]-[0007], [0031]-[0035], [0047]-[0049].
White et al. teaches the missing limitation. White et al. states that a “render graph asset” is a digital file and “is represented as a JavaScript Object Notation (JSON) data file.” White et al. further teaches that “the render graph asset pipeline 506 then generates render graph file 508 for each render graph,” and that a component may have “an render graph asset handle (e.g., an identifier) to render graph file 508,” such that “at runtime ... the camera component 502 is able to provide its own render graph rather than a default one.” This teaches determining a configuration file having a preset binding relationship with a selected processing pipeline. White et al. also teaches that “the render graph file 508 is a JSON file that defines the collection of nodes, render targets, and the connections between the nodes and render targets,” and that “the render graph API compiles the render graph file 508 to generate render graph data object 510.” This teaches reading the configuration information from the configuration file. — White et al., ¶¶ [0016]-[0017], [0031]-[0036].
Before the effective filing date of the claimed invention, a person of ordinary skill in the art would have been motivated to combine Kim et al. with White et al. because Kim et al. teaches an image-processing architecture that selects processing modes and performs neural-network-based image processing, while White et al. teaches using a file-based configuration mechanism, such as a JSON render graph file, to bind a selected processing pipeline to a specific configuration and load its node / connection information at runtime. A POSITA would have found it obvious to implement Kim et al.’s selected image-editing pipeline using White et al.’s configuration-file approach in order to improve modularity, configurability, reuse, and predictable runtime selection of the appropriate processing pipeline.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Claim 19 does not include any additional limitations that would significantly distinguish it from claim 5. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Ground of Rejection 4
Claim 7 is rejected under 35 U.S.C. § 103 as being unpatentable over Kim et al. (US20210006755A1) in view of Chang et al. (US20190251674A1), and further in view of Lugaresi et al. (NPL).
As per Claim 7, Kim alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Chang and Lugaresi, they collectively teach all of the limitation(s).
Kim, Chang, and Lugaresi teach Claim 7, which recites:
“The image processing method according to claim 1, wherein: the pre-processing function node comprises one or more of an image transformation node, a region detection node, or a region image cropping node; and the post-processing function node comprises one or more of the image transformation node or a time-domain smoothing node.”
Kim et al. teaches the base image-processing method of claim 1 and teaches pre-processing and post-processing function nodes generally. Kim et al. teaches a “pre-processor 100” and “main processor 300,” and teaches pre-processing and post-processing modules such as “an X-talk correction module 120 a and a despeckle module 130 a,” and “a Bayer demosaic module 310 a, a Bayer denoising module 320 a, and a sharpening module 330 a.” Kim et al. thus teaches image-transformation type nodes generally, but Kim et al. does not clearly teach the specific claimed node examples of region detection node, region image cropping node, and time-domain smoothing node.
Chang et al. teaches the missing pre-processing node types. Chang et al. states that “face parser 105 processes an input photo to automatically detect regions of the photo that include a subject’s skin,” which teaches a region detection node. Chang et al. also teaches that “frequency separator 115 ... separates the skin mask into a high frequency layer and a low frequency layer,” which teaches an image transformation node. Chang et al. further teaches that “patch segmenter 130 separates the high frequency layer into a number of smaller patches” and “can segment the high frequency layer into a grid of patches ... or any other collection of regions,” which teaches a region image cropping node.
Lugaresi et al. teaches the missing post-processing node types. Lugaresi et al. teaches that, in the face-landmark and segmentation pipeline, “the landmarks and masks are temporally interpolated across frames,” and “temporal re-sampling” may be used, which teaches a time-domain smoothing node. Lugaresi et al. also teaches that “the annotations from the two tasks are overlaid onto the camera frames,” which teaches a post-processing image transformation node.
Before the effective filing date of the claimed invention, a person of ordinary skill in the art would have been motivated to combine Kim et al. with Chang et al. and Lugaresi et al. because Kim et al. teaches an image-processing architecture having pre-processing, neural-network-based processing, and post-processing; Chang et al. teaches concrete pre-processing nodes for effect-editing workflows, including region detection, image transformation, and patch-based cropping; and Lugaresi et al. teaches concrete post-processing nodes, including temporal interpolation / re-sampling and annotation overlay on frames. A POSITA would have found it obvious to use these known node types within Kim et al.’s configurable image-processing pipeline in order to improve modularity, enable richer image-editing workflows, and obtain predictable benefits in preprocessing accuracy and postprocessing smoothness.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Ground of Rejection 5
Claim 9 is rejected under 35 U.S.C. § 103 as being unpatentable over Kim et al. (US20210006755A1) in view of Rymkowski et al. (US20180082407A1).
As per Claim 9, Kim alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Rymkowski, they collectively teach all of the limitation(s).
Kim and Rymkowski teach Claim 9, which recites:
“The image processing method according to claim 1, wherein operation of the effect editing performed on the initial image is triggered by triggering an effect control on an interface.”
Kim et al. teaches the base image-processing method of claim 1 and teaches an interface that receives user input. Kim et al. states that the image processing device includes a “user interface 2700” and that “The user interface 2700 may receive a user input and provide, to the AP 2100, a signal corresponding to the received user input.” However, Kim et al. does not clearly teach that the claimed effect editing operation is triggered by an effect control on that interface. — Kim et al., ¶ [0089].
Rymkowski et al. teaches the missing limitation. Rymkowski et al. states that “the process may begin at Step 402 by obtaining an artistic style to use” and that “a user may simply select the artistic style that he or she wishes to apply, e.g., from a list of paintings, artists, or predetermined available artistic style ‘filters.’” This teaches that operation of the effect editing is triggered by triggering an effect control on an interface. Rymkowski et al. also teaches a device including a “display” and “user interface.” — Rymkowski et al., ¶¶ [0042], [0052]-[0054].
Before the effective filing date of the claimed invention, a person of ordinary skill in the art would have been motivated to combine Kim et al. with Rymkowski et al. because Kim et al. teaches an image-processing architecture having a user interface for receiving input, while Rymkowski et al. teaches initiating an image effect by user selection of an artistic style / filter through the interface. A POSITA would have found it obvious to use Rymkowski et al.’s interface-triggered effect selection in Kim et al.’s image-processing system to improve usability and provide predictable user-controlled activation of image effects.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Ground of Rejection 6
Claim 11 is rejected under 35 U.S.C. § 103 as being unpatentable over Kim et al. (US20210006755A1) in view of Chang et al. (US20190251674A1).
As per Claim 11, Kim alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Chang, they collectively teach all of the limitation(s).
Kim and Chang teach Claim 11, which recites:
“The image processing method according to claim 7, wherein: the processing logic corresponding to the image transformation node is logic configured to pre-process the initial image; the processing logic corresponding to the region detection node is logic configured to determine a position of a target object in the initial image; and the processing logic corresponding to the region image cropping node is logic configured to extract the target object from the initial image.”
Kim et al. teaches the base image-processing method of claim 1 and generally teaches image transformation in pre-processing, e.g., image-processing modules and neural-network-based transformation of image data. However, Kim et al. does not clearly teach the specific claimed logic of region detection to determine a target-object position and region image cropping to extract the target object from the initial image. — Kim et al., ¶¶ [0037]-[0040], [0044], [0047].
Chang et al. teaches the missing limitations. Chang et al. states that “frequency separator 115 generally separates an input photo into a high frequency layer and a low frequency layer,” which teaches the image transformation node as logic configured to pre-process the initial image. Chang et al. also states that “face parser 105 processes an input photo to automatically detect regions of the photo that include a subject’s skin” and “identify the various parts of a subject’s face,” which teaches the region detection node as logic configured to determine a position of a target object in the initial image. Chang et al. further states that “skin filter 110 ... filters out the non-skin regions ... [and] produces ... a skin mask comprising the identified and filtered skin regions,” which teaches the region image cropping node as logic configured to extract the target object from the initial image. Chang et al. also teaches that “patch segmenter 130 separates the high frequency layer into a number of smaller patches,” further supporting region extraction / cropping. — Chang et al., ¶¶ [0032], [0034], [0037], [0039]-[0040].
Before the effective filing date of the claimed invention, a person of ordinary skill in the art would have been motivated to combine Kim et al. with Chang et al. because Kim et al. teaches a configurable image-processing architecture with pre-processing and neural-network-based image operations, while Chang et al. teaches specific pre-processing logic for image transformation, region detection, and region extraction applied to an input image before retouching. A POSITA would have found it obvious to use Chang et al.’s region-based pre-processing operations within Kim et al.’s image-processing pipeline in order to improve preprocessing accuracy, isolate the target object or region for later inference, and obtain predictable image-editing results.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Conclusion
The prior art made of record and relied upon in this action is as follows:
Patent Literature:
Rymkowski et al. (US20180082407A1) — “Style transfer-based image content correction.”
White et al. (US20200104970A1) — “Customizable Render Pipelines using Render Graphs.”
Chang et al. (US20190251674A1) — “Deep-learning-based automatic skin retouching.”
Kim et al. (US20210006755A1) — “Image processing device including neural network processor and operating method thereof.”
Non-Patent Literature (NPL):
Lugaresi et al. (NPL) — “MediaPipe: A Framework for Building Perception Pipelines”, 2019. Available at: [https://3dvar.com/Lugaresi2019MediaPipe.pdf]
Jiang et al. (NPL) — “Mainstream: Dynamic Stem-Sharing for Multi-Tenant Video Processing”, 2018. Available at: [https://www.usenix.org/system/files/conference/atc18/atc18-jiang.pdf]
Note: A PDF copy of each NPL reference is attached with this Office Action. URLs are included for applicant convenience. If a link becomes unavailable in the future, the citation information may be used to locate the reference or access archived versions via the Wayback Machine.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure and is listed as follows:
Patent Literature:
Golden et al. (US20180259608A1) — “Automated cardiac volume segmentation.”
Non-Patent Literature (NPL):
(none)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ADEEL BASHIR whose telephone number is (571) 270-0440. The examiner can normally be reached Monday-Thursday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached on (571) 276-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ADEEL BASHIR/
Examiner, Art Unit 2616
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616
Prosecution Insights