Detailed Action
1. Claims 1-20 are pending in this Application.
Notice of Pre-AIA or AIA Status
2. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103, which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
3. Claims 1,5-8, 11, 15-16 and 20 are rejected under 35 U.S.C. 103(a) as being unpatentable over KR-20220036061, pub. 03/22/2022, in views CHOI JAE YOUNG (hereafter CHOI, US-20110279640, pub., 11/17/2011.
As to claim 1, KR-20220036061 teaches An electronic device comprising: memory; a camera module; a communication module; and at least one processor operatively connected to the memory, the camera module and the communication module, wherein the memory, when executed by the at least one processor, cause the electronic device (Abstract, claim 1, an electronic devices includes , a memory in which a downscaling network of the first artificial intelligence model is stored; communication interface; and a processor connected to the memory and the communication interface to control the electronic device,) to: establish a call connection with a network based on the communication module (Figs . 6 and 7), identify a first image (claim 1, The processor identify an input image and input the input image into the downscaling network) captured based on the camera module, identify first information associated with a first bitrate corresponding to the first image, based on a communication environment between the network and the electronic device, identify a second image corresponding to the first image output from an artificial intelligence model for down-scaling, trained to receive information associated with a high-resolution image (claim 1, Figs.5-7, Abstract, the processor, Input an input image into the downscaling network to obtain an output image in which the input image is downscaled)and transmit the second image; output a low-resolution image, by inputting the first image and the first information to the artificial intelligence model (Claim 1, [0063] That is, the processor (130) can lower the resolution of the input image through a downscaling network and transmit it to the second electronic device (200) along with additional information, thereby reducing the amount of data transmitted);
however, it is noted that KR-20220036061 does not specifically teach “the camera module, establish a call connection with a network based on the communication module, identify a first image captured based on the camera module, identify first information associated with a first bitrate corresponding to the first image, based on a communication environment between the network and the electronic device”
On the other hand CHOI teaches establish a call connection with a network based on the communication module, identify a first image captured based on the camera module, identify first information associated with a first bitrate corresponding to the first image, based on a communication environment between the network and the electronic device (abstract, Fig.3, [0024] claim 7, the display apparatus according to an embodiment includes: a camera for acquiring video data and outputting the video data at a predetermined resolution; a network monitoring unit for detecting an available bandwidth of a network for transmitting video communication data; a resolution adjusting unit for adjusting the output resolution of the camera in accordance with the detected available bandwidth; and a communication unit for transmitting video data outputted from the camera to an external device. Further as shown in Fig.13, the size of the video call image is adjusted (reduced) and the reduced portion of the video image is transmitted)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate a method of detecting an available bandwidth of a network for transmitting video communication data, and adjusting the output resolution of the camera in accordance with the detected available bandwidth taught by CHOI (see [0002]) into KR-20220036061.
The suggestion/motivation for doing so would have been allows user of KR-20220036061 to guarantee the transmission of video data in remote location by adjusting the size of a video image and transmitting the adjusted video image data based on the availably of bandwidth.
Regarding claim 11 the combination of KR-20220036061 and CHOI teaches all the limitation of claim 11, as discuss in claim 1 above , except the underline section of the limitation: “display module to display at least a portion of the second image” . However, CHOI still teach this limitation. Specifically, CHOI teaches( Fig.13, [0024] as shown in Fig.13, the size of the video call image is adjusted (reduced) and the reduced portion of the video image is transmitted).
As to claim 20, KR-20220036061 taches One or more non-transitory computer-readable storage media storing at least one computer-readable instruction that, when executed by at least one processor of an electronic device, configures the electronic device to perform operations ([0113], the method according to the various embodiments described above may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed); regarding the remaining limitation of claim 20, all the remining claim limitations are set forth and rejected as per discussion for claim 1.
As to claim 5, KR-20220036061 teaches the memory, when executed by the at least one processor, cause the electronic device to, as at least part of transmitting the second image (this limitations discussed in claim 1 above): generate a bitstream by encoding the second image, and transmit the bitstream through the call connection([0110], The processor inputs an input image to the downscaling network to obtain an input image. acquires a down-scaled output image and controls a communication interface to transmit the output image to another electronic device. The processor 130 may obtain AI meta information corresponding to the setting value through the AI meta generator and transmit the encoded image and AI meta information to the second electronic device 200 through the stream generator.)
As to claim 15, CHOI teaches the memory, when executed by the at least one processor, cause the electronic device to, as at least part of receiving the first image (this limitations discussed in claim 1 above): receive a bitstream through the call connection ([0110], the processor 130 may obtain AI meta information corresponding to the setting value through the AI meta generator and transmit the encoded image and AI meta information to the second electronic device 200 through the stream generator), and identify the first image by decoding the bitstream (Abstract, [0049], Additionally, the second electronic device 200 may decode the received image and upscale the decoded image through an artificial intelligence model.).
As to claim 6, CHOI teaches the memory, when executed by the at least one processor, cause the electronic device to identify the communication environment based on at least one of a one-way delay, a perceived bitrate, a packet loss rate, or a bandwidth (claims1 and 7, Abstract, the method of controlling a DTV according to claim 1, further comprising: displaying, on a screen of the DTV, information on the detected available
bandwidth and information on a new output resolution of the camera to which the camera can adjust based on the detected available bandwidth).
Regarding claim 16, all claim limitations are set forth and rejected as per discussion for claim 6.
As to claim 7, CHOI teaches the artificial intelligence model for down-scaling includes, a first portion extracting a feature of the first image, a second portion extracting a feature of the first information, a multiplier cross-multiplying the feature of the first image and the feature of the first information, a third portion for enhancing a result of the cross-multiplying by the multiplier and configuring a residual image, a fourth portion for down-scaling the first image, and an adder for adding an output result of the third portion and an output result of the fourth portion, and wherein a result of adding by the adder is provided as the second image (Claim 1, [0054], The first artificial intelligence model may include a downscaling network and an upscaling network. The downscaling network and the upscaling network of the first artificial intelligence model can be trained based on a sample image, a first intermediate image obtained by inputting the sample image into the downscaling network, a first final image obtained by inputting the first intermediate image into the upscaling network of the first artificial intelligence model, a second intermediate image obtained by downscaling the sample image by a legacy scaler, and a second final image obtained by upscaling the first intermediate image by a legacy scaler. In addition as shown in Figs. 1,5-7, the neural network is designed to train a plurality of images and portion images)
As to claim 8, CHOI teaches wherein the artificial intelligence model for down-scaling is a ResNet, wherein the first portion includes at least one convolution layer, wherein the second portion is a DenseNet, wherein the third portion includes at least one convolution layer, and wherein the fourth portion is a Bicubic down scaler (Figs.5-7, [0004]Figs5-7 illustrates the neural network has scale- up and scale convolution layer down- down- convolution layer. The ResNet, DenseNet and Bicubic down scaler are a well-known structure, and corresponds to the artificial intelligence model performing downscaling and the artificial intelligence model performing upscaling can be trained as one artificial intelligence model, as illustrated in Fig. 1b ).
4. Claims 9-10 and 17-19 are rejected under 35 U.S.C. 103(a) as being unpatentable over
KR-2022003606 in views CHOI, US-20110279640, further in view of Reddy et al., (hereafter Reddy), US20200226718 A1, pub. 07/16/2020
As to claim 17, KR-20220036061 teaches the memory, when executed by the at least one processor, cause the electronic device to: identify training data including a first image, which is a high-resolution image, and first information associated with a bitrate, identify a second image, which is a low-resolution image (this limitation discussed in claim 1 above, see claim 1, Fig. 5-7 ), output from a first AI model for down-scaling, based on inputting the first image and the first information to the first AI model (Abstract, Figs. 5-7. calim1, The processor, Input an input image into the downscaling network to obtain an output image in which the input image is downscaled), identify a third image, which is a high-resolution image, output from a second AI model for up-scaling, based on inputting the second image and the first information to the second AI model, identify a fourth image by down-scaling the first image(this limitations also discussed in claim 1 above. As shown in Figs. 5-7, the neural network designed to receive and process a plurality of images);
however, it is noted that KR-20220036061 does not specifically teach “identify a total loss based on a first loss corresponding to the first image and the third image and a second loss corresponding to the second image and the fourth image, and train at least a portion of the first AI model and the second AI model based on the total loss”
On the other hand Reddy teaches identify a total loss based on a first loss corresponding to the first image and the third image and a second loss corresponding to the second image and the fourth image, and train at least a portion of the first AI model and the second AI model based on the total loss([0016], techniques for training a machine learning model to perform super-resolution on images using high-frequency loss information. The techniques involve generating a plurality of degraded images from a plurality of reference images. The degraded images are then propagated through a machine learning model, which learns/generates one or more mapping functions to produce the reference images from the corresponding degraded images. Multiple iterations of learning/training are performed until the machine learning system is determined to be adequately accounting for high frequency losses. A pixel loss for the training process is determined based on pixel intensity differences between the enhanced images and the reference images, and a total loss is calculated as a sum of the pixel loss and the high-frequency loss. The total loss is backpropagated to the machine learning model, which uses the loss information to determine whether further updates to the mapping function are to be performed (e.g., to further reduce the loss). Another iteration of the learning process is initiated accordingly).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate a method of a neural network that can be trained to perform denoising and/or deblurring processes taught by Reddy (see [0002]) into modified KR-20220036061.
The suggestion/motivation for doing so would have been allows user of modified KR-20220036061 to predict a high-resolution image from a low-resolution version based on the training data.
Regarding claim 9, all claim limitations are set forth and rejected as per discussion for claim 17.
As to claim 18, KR-20220036061 teaches the memory, when executed by the at least one processor, cause the electronic device to: identify a fifth image, which is a low-resolution image, output from the first AI model, based on inputting the first image and the first information to the first AI model, identify a sixth image by encoding the fifth image and decoding a result of the encoding, identify a seventh image, which is a high-resolution image, output from the second AI model, based on inputting the sixth image and the first information to the second AI model , identify an eighth image obtained by enhancing the first image (All the limitations are similar to the limitations of claim 17. In addition As shown in Figs. 5-7, the neural network designed to receive and process a plurality of images ),
however, it is noted that KR-20220036061 does not specifically teach “identify a total loss based on a first loss corresponding to the first image and the third image and a second loss corresponding to the second image and the fourth image, and train at least a portion of the first AI model and the second AI model based on the total loss”
On the other hand Reddy teaches identify a total loss based on a first loss corresponding to the first image and the third image and a second loss corresponding to the second image and the fourth image, and train at least a portion of the first AI model and the second AI model based on the total loss ([0016], ([0006], [0027] , Multiple iterations of learning/training are performed until the machine learning system is determined to be adequately accounting for high frequency losses. A pixel loss for the training process is determined based on pixel intensity differences between the enhanced images and the reference images, and a total loss is calculated as a sum of the pixel loss and the high-frequency loss. The total loss is backpropagated to the machine learning model, which uses the loss information to determine whether further updates to the mapping function are to be performed (e.g., to further reduce the loss). Another iteration of the learning process is initiated accordingly)
Regarding claim 10, all claim limitations are set forth and rejected as per discussion for claim 18.
As to claim 19, Reddy teaches a second loss corresponding to the second image and the fourth image is a loss between images obtained by enhancing the second image and the fourth image([0006], [0027], The determined pixel loss value can represent the difference between the pixel values of the several locations of the reference images 110 and the pixel values of the associated locations of the corresponding predicted images 140. For example, assume that a first reference image 110 is processed to generate a corresponding first degraded image 120, which is then processed to generate corresponding first predicted image 140. Predicted images using a mapping function associated with a machine learning process to at least partially remove the one or more first degradations).
Allowable Subject Matter
5. Claims 2-4 and 12-14 are objected to as being dependent upon a rejected base claims but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claim.
6. Regarding dependent claims 2 and 12 no prior art is found to anticipate or render the following limitation obvious:
“Wherein the memory, when executed by the at least one processor, cause the electronic device to, as at least part of identifying the first information associated with the first bitrate corresponding to the first image: identify a first bit per pixel (BPP), obtained by dividing the first bitrate by a product of a first frame rate associated with the first image and a resolution associated with the first image, as the first information.”
7. Claims 3-4 and 13-14 are objected since they are depending directly or in directly on the objected claim 2 and 12 respectively
Prior art of record but not applied in the rejection
“TRAINING AND UTILIZING AN IMAGE EXPOSURE TRANSFORMATION NEURAL NETWORK TO GENERATE A LONG-EXPOSURE IMAGE FROM A SINGLE SHORT-EXPOSURE IMAGe”, US 20190333198 A1, pub. 10/312019. to Wang. et al., disclosed
In addition, the term “image exposure transformation network” refers to a neural network trained to generate a long-exposure image from a short-exposure image. In one or more embodiments, an image exposure transformation network includes an image exposure generative adversarial network (“GAN”). The GAN employs adversarial learning to generate realistic synthesized long-exposure images. In particular, the GAN includes an image exposure generator neural network (“generator neural network”) that a learns to generate a synthesized long-exposure image from a single long-exposure image and a corresponding optical input flow, and in some cases, an attention map. The GAN also includes an adversarial discriminator neural network (“discriminator neural network”) that learns to distinguish realistic long-exposure images from non-realistic long-exposure images, (see[ 0053]) .
As used herein, the term “adversarial learning” refers to a machine-learning algorithm (e.g., the GAN) where opposing learning models are learned together. In particular, the term “adversarial learning” includes solving a plurality of learning tasks in the same model (e.g., in sequence or in parallel) while utilizing the roles and constraints across the tasks. In some embodiments, adversarial learning includes minimizing total loss between one or more loss terms, as further described below(see [0054]).
Contact Information
Any inquiry concerning this communication or earlier communication from the examiner should be directed to Mekonen Bekele whose telephone number is (469) 295-9077.The examiner can normally be reached on Monday -Friday from 9:00AM to 6:50 PM Eastern Time.
If attempt to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Eng, George can be reached on (571) 272-7495.The fax phone number for the organization where the application or proceeding is assigned is 571-237-8300. Information regarding the status of an application may be obtained from the patent Application Information Retrieval (PAIR) system. Status information for published application may be obtained from either Private PAIR or Public PAIR.
Status information for unpublished application is available through Privet PAIR only.
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have question on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866.217-919 (tool-free)
/MEKONEN T BEKELE/Primary Examiner, Art Unit 2699