DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 1 and 7 are objected to because of the following informalities:
Claim 7 line 4 cites the following: “a processing unit configured to redirect the first image to obtain a second image”. The term redirect makes it unclear how the first image is used to obtained the second image. The term also does not clarify how it relates to the coordinate system of the second and target image. For examination purposes, Examiner has interpreted this limitation to mean that a coordinate transformation process is performed on the first image to match the target image thereby producing a second image with a consistent coordinate system to the target image. Claim 1 recites a similar limitation.
Claims 7 line 5-6 cites the following “coordinate system of the target image”. It is unclear whether or not the target image refers to the annotated target image. For examination purposes, the Examiner will interpret that the “target image” does refer to the “annotated target image” as it would be a lack of antecedent basis if “the target image” does not refer to “an annotated target image”. Claim 1 recites a similar limitation.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 14 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because claim 6 is directed to a signal per se.
Claim 14 recites a computer-readable medium. The broadest reasonable interpretation of a claim drawn to a computer readable medium (also called machine readable medium and other such variations) typically covers forms of non-transitory tangible media and transitory propagating signals per se in view of the ordinary and customary meaning of computer readable media, particularly when the specification is silent. See MPEP 2111.01. When the broadest reasonable interpretation of a claim covers a signal per se, the claim must be rejected under 35 U.S.C. 101 as covering non-statutory subject matter. The USPTO recognizes that applicants may have claims directed to computer readable media that cover signals per se, which the USPTO must reject under 35 U.S.C. 101 as covering both non-statutory subject matter and statutory subject matter. A claim drawn to such a computer readable medium that covers both transitory and non-transitory embodiments may be amended to narrow the claim to cover only statutory embodiments to avoid a rejection under 35 U.S.C. $ 101 by adding the limitation "non-transitory" to the claim. Such an amendment would typically not raise the issue of new matter, even when the specification is silent because the broadest reasonable interpretation relies on the ordinary and customary meaning that includes signals per se.
Claim 15 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because the claimed invention is directed to a software per se.
Claim 15 recites a computer program product. Claimed computer programs do not define any structural and functional interrelationships between the computer program and other claimed elements of a computer which permit the computer program’s functionality to be realized. In contrast, a claimed non-transitory computer-readable storage medium encoded with a computer program is a computer element which defines structural and functional interrelationships between the computer program and the rest of the computer which permit the computer program’s functionality to be realized, and is thus statutory. Accordingly, it is important to distinguish claims that define descriptive material per se from claims that define statutory inventions.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3, 7, 9, and 13-16, is/are rejected under 35 U.S.C. 103 as being unpatentable over Zou et al (CN 110490960 A) and Ma et al (US 20150138399 A1), hereinafter Zou and Ma respectively.
Regarding claim 7, Zou teaches a data acquisition apparatus, comprising: a shooting unit (“A real-shot image acquisition module is used to acquire real-shot images of the target scene” – Par 19) configured to photograph an annotated target image to obtain a first image;
“The composite image generation module is used to reshape the style of the rendered image based on the real-shot image to obtain a composite image, and to determine the annotation information of the rendered image as the annotation information of the composite image.”- Par 20
NOTE: Zou discloses in Par 48-54 steps S101-S104. A rendered image of a simulation scene of a target scene is obtained, annotation information is generated to annotate/mark where the object of interest is located in the image. A real-shot image acquisition module is used to capture an image of the rendered image which then goes through a composite image generation module to create a new composited image of the target image with the annotated information which functionally corresponds to obtaining a first image of an annotated target image.
Note: Zou Paragraph 9 teaches “this method utilized 3D rendering technology to render multiple initial labeled images based on real world image, Zou paragraph 53 further teaches real shot images can be actual images of the target scene captured on-site. Therefore, the real word object disused in paragraph 9 can obviously be actual image being captured (photographed) as photographic is well know in the art as being the fastest and efficient way of getting real world image.
Note: The rendered initial labeled image based on the real world image of Zou is the first image.
and a transferring unit configured to perform style transfer on the first image according to a third image to obtain an annotated raw image, wherein the third image is obtained by photographing a real environment corresponding to the target image.
“To further improve the realism of the final synthesized image, reduce the style domain difference between the synthesized image and the real-world image, and ensure that the style of the synthesized image and the real-world image are as close as possible, the image style transfer process is as shown in Figure 3. In step S104, based on the acquired real-world image, the generated rendered image undergoes style reshaping to obtain the synthesized image, and the determined annotation information is used as the annotation information for the synthesized image.” – Par 108
NOTE: Zou paragraph 0019 disclosed “a real shot image acquisition module is used to acquire real shot images of the target scene,
Zou paragraph 20 disclosed “The composite image generation module is used to reshape the style of the rendered image based on the real-shot image to obtain a composite image, and to determine the annotation information of the rendered image as the annotation information of the composite image.”
Note: the real shot image of the target scene of paragraph 19 and 20 of Zou is the third image
Zou teaches a transferring unit but Zou does not teach a processing unit configured to redirect the first image to obtain a second image, wherein a coordinate system of the second image is consistent with a coordinate system of the target image and that the transferring unit is configured to perform style transfer on the second image according to a third image to obtain an annotated raw image. However Ma teaches, a processing unit configured to redirect the first image to obtain a second image, wherein a coordinate system of the second image is consistent with a coordinate system of the target image;
“In one embodiment, and based on the results of the text block matching described above, bounding boxes of the text lines in the best match are used to estimate an affine or homograph transform matrix, also referred to herein as a "first transformation matrix." The first transform matrix is applied to every pixel in the second image (test frame) to transform the second image to coordinate system in the first image (reference frame). In this way, the second image is adjusted to the first image plane, and a composite image including information depicted in both the two images is derived.” – Par 96
NOTE: Ma teaches a processing unit 210 configured to transform a second image’s coordinate system to match that of a first image’s coordinate system using an affine transform matrix technique. This functionally corresponds to a coordinate of the second system being consistent with a coordinate system of the target image. After the combination, the transforming of the coordinate system of one image to match with another can modify the first image of the annotated target image obtained by Zou so that when creating the second image, the coordinate system will be consistent with the image of the annotated target image. Ma discloses that the techniques of their invention may be implemented in hardware utilizing one or more processors such as processing unit 210, see par 62.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Zou by incorporating the teachings of Ma to have a processing unit configured to redirect the first image to create a second image wherein the second image has a coordinate system consistent with the target image. One would be motivated to make this combination since the second image created by the processing unit would be an improved edited version of the first image that better represents the annotated target image. This will then allow the transferring unit to use the improved second image for the style transferring process in order to improve the realism of the final synthesized image and reduce domain difference between the synthesized and real-world image. NOTE: Zou discloses a style transfer process with the adversarial network model to transfer the style image of a real-shot image to a generated rendered image to generate a synthetic image that preserves the annotated information. The style transfer model disclosed by Zou does not explicitly teach that the style transfer is performed on the second image using the third image as reference.
After the combination, generating the second image of the annotated target image as taught by Ma can substitute the generated rendered image as taught by Zou so that the modification will allow transferring the image style from the real shot image of the scene (third image as previously discussed) obtained by Zou to the second image of Zou as modified by Ma.
Regarding claim 1, the claim recites similar limitations to claim 7. Therefore, method claim 1 corresponds to the apparatus disclosed in claim 7 and is rejected for the same reasons of obviousness as used above.
Regarding claim 13, the claim recites similar limitations to claim 7. Therefore, device claim 13 corresponds to the apparatus disclosed in claim 7 and is rejected for the same reasons of obviousness as used above.
Regarding claim 14, the claim recites similar limitations to claim 7. Therefore, computer-readable storage medium claim 14 corresponds to the apparatus disclosed in claim 7 and is rejected for the same reasons of obviousness as used above.
Regarding claim 15, the claim recites similar limitations to claim 7. Therefore, computer program product claim 15 corresponds to the apparatus disclosed in claim 7 and is rejected for the same reasons of obviousness as used above.
Regarding claim 16, Zou in view of Ma teaches the method of claim 1. Zou further teaches a chip system, applied to an electronic device,
“The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions.” – Zou Par 251
wherein the chip system comprises one or more interface circuits and one or more processors;
“In a typical configuration, a computing device includes one or more processors (CPU), input/output interfaces, network interfaces, and memory.” – Zou Par 257
NOTE: Zou in view of Ma teaches a processing unit processing unit configured to redirect the first image to obtain a second image, wherein a coordinate system of the second image is consistent with a coordinate system of the target image; and a transferring unit configured to perform style transfer on the second image according to a third image to obtain an annotated raw image, wherein the third image is obtained by photographing a real environment corresponding to the target image which functionally corresponds to an interface circuit and would also inherently require a chip system applied to an electronic device. Please see rejection of claim 7. The processing unit and transferring unit would inherently require a chip system applied to an electronic device to carry out its functions.
the interface circuit and the processor are interconnected through lines; the interface circuit is configured to receive a signal from a memory of the electronic device and send the signal to the processor, wherein the signal comprises a computer instruction stored in the memory;
“Furthermore, the processor 801 may be configured to communicate with the memory 802 and execute a series of computer-executable instructions in the memory 802 on the composite image generation device.” – Zou Par 172, Lines 6-7
NOTE: To carry the data acquisition method of claim 1, the computer instructions must be stored in a memory which is sent to an interface circuit to then send a signal to the processor in order for the functions to be executed. This structure would be necessary to perform the functions of the processing unit and transferring unit as disclosed by Zou in view of Ma.
and when the processor executes the computer instruction, the electronic device executes the data acquisition method of claim 1
NOTE: Zou in view of Ma teaches the data acquisition of method claim 1. Please see rejection of claim 1.
Regarding claim 9, Zou in view of Ma teaches the data acquisition apparatus of method 7. Zou further teaches wherein the transferring unit is configured to: obtain a generator and a discriminator according to the second image and the third image
“Specifically, in the style transfer process, the deep neural network selected is an adversarial neural network. The adversarial neural network trains both the generator and the discriminator. The generator aims to make the style-transferred image as realistic as possible, while the discriminator aims to distinguish between the real sample image and the style-transferred image. In this way, the generator and the discriminator compete with each other during the training of the adversarial network model for style transfer, achieving a mutually reinforcing effect. When the performance of the generator and the discriminator is dynamically balanced, the model training is considered complete, and the preset adversarial network model containing the iteratively optimized generator and discriminator is determined as the trained adversarial network model.” – Par 116
and perform style transfer on the second image according to the generator and the discriminator to obtain the annotated raw image.
“This adversarial network model is obtained by continuously optimizing the generator and discriminator in the initial adversarial network model through dynamic competition between the generator and discriminator based on the sample images. This can reduce the image style domain difference between the synthesized images and the real-world images, and ensure that the image styles of the synthesized images are as close as possible to those of the real-world images.” – Par 110, Lines 2-5
NOTE: Zou discloses that the adversarial network model used for style transfer comprises a generator and discriminator. The discriminator is optimized to identify synthesized images produced by the generator from real images while the generator is optimized to create realistic synthesized images that cannot be detected by the discriminator, see par 114. This adversarial network model will then be able to perform style transfer taught by Zou can have the second image generated by Ma and the third image of a real environment obtained by Zou to create an annotated raw image as descried in the rejection of claim 7.
Regarding claim 3, the claim recites similar limitations to claim 9. Therefore, method claim 1 corresponds to the apparatus disclosed in claim 9 and is rejected for the same reasons of obviousness as used above.
Claim(s) 2, 8, and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zou, Ma and Li et al (US 20220101548 A1), hereinafter Li.
Regarding claim 8, Zou in view of Ma teaches the apparatus of claim 7. Zou does not teach wherein the processing unit is configured to: establish a first coordinate system according to the target image and a preset checkerboard-marked image, and obtain coordinates of corner points in the preset checkerboard-marked image in the first coordinate system; photograph the preset checkerboard-marked image, and obtain coordinates of the corner points in the preset checkerboard-marked image in a second coordinate system, wherein the second coordinate system is a coordinate system established according to the photographed image; obtain a transformation matrix between the second coordinate system and the first coordinate system according to the coordinates of the corner points in the preset checkerboard-marked image in the first coordinate system and the coordinates of the corner points in the preset checkerboard-marked image in the second coordinate system; and perform affine transformation on the first image according to the transformation matrix between the second coordinate system and the first coordinate system to obtain the second image. However, Li teaches wherein the processing unit is configured to: establish a first coordinate system according to the target image and a preset checkerboard-marked image, and obtain coordinates of corner points in the preset checkerboard-marked image in the first coordinate system;
“The calibration of the photographic camera is implemented by Zhang's calibration method, in which first a photographic camera coordinate system and a world coordinate system are set, and using an image of the photographic camera on a checkerboard placed at a preset position, two-dimensional coordinates of corner point positions of the checkerboard in the image are calculated, and then a transformation matrix is calculated with real three-dimensional coordinates of the corner points of the checkerboard; ” – Par 79, Lines 1-8
NOTE: Li teaches implementing “Zhang’s calibration method” which comprises of using a preset checkerboard image for a photograph camera coordinate system and a world coordinate system. Li explicitly teaches calculating the corner points for a photographic camera coordinate system which can be understood as the corner coordinates in the second coordinate system. This implementation would also have to be used in order to obtain the coordinate system and the corner points of the world coordinate system which can be understood as the first coordinate system.
photograph the preset checkerboard-marked image, and obtain coordinates of the corner points in the preset checkerboard-marked image in a second coordinate system, wherein the second coordinate system is a coordinate system established according to the photographed image;
“using an image of the photographic camera on a checkerboard placed at a preset position, two-dimensional coordinates of corner point positions of the checkerboard in the image are calculated” – Par 79, Lines 5-8
NOTE: Li discloses a photographic camera that takes an image of the preset checkerboard which is then used to find the corner position coordinates of a second coordinate system.
obtain a transformation matrix between the second coordinate system and the first coordinate system according to the coordinates of the corner points in the preset checkerboard-marked image in the first coordinate system and the coordinates of the corner points in the preset checkerboard-marked image in the second coordinate system;
“In one embodiment, and based on the results of the text block matching described above, bounding boxes of the text lines in the best match are used to estimate an affine or homograph transform matrix, also referred to herein as a "first transformation matrix." The first transform matrix is applied to every pixel in the second image (test frame) to transform the second image to coordinate system in the first image (reference frame). In this way, the second image is adjusted to the first image plane, and a composite image including information depicted in both the two images is derived.” – Ma Par 96
NOTE: Ma discloses a transformation matrix between a first and second coordinate system but does not explicitly teach that the transformation matrix is done between a first and second coordinate system according to the coordinates of the corner points. However, After the combination, the transformation process taught by Ma can be done on the first world coordinate system and second photographic camera coordinate system obtained by the methods of Li. Since the transform matrix is performed pixel-by-pixel in Ma, the corner coordinates as obtained by Li will be naturally included in the transformation matrix.
and perform affine transformation on the first image according to the transformation matrix between the second coordinate system and the first coordinate system to obtain the second image.
“In one embodiment, and based on the results of the text block matching described above, bounding boxes of the text lines in the best match are used to estimate an affine or homograph transform matrix, also referred to herein as a "first transformation matrix." The first transform matrix is applied to every pixel in the second image (test frame) to transform the second image to coordinate system in the first image (reference frame). In this way, the second image is adjusted to the first image plane, and a composite image including information depicted in both the two images is derived.” – Ma Par 96
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Zou by incorporating the teachings of Li to obtained corner coordinate points in a preset checkerboard image in a first and second coordinate system, obtain a transformation matrix between the two coordinate systems based on the coordinate points and perform affine matrix transformation matrix between the second coordinate system and the first coordinate system to obtain the second image. One would be motivated to make this combination to accurately map the annotation locations between the target image and the photographed image.
Regarding claim 2, the claim recites similar limitations to claim 8. Therefore, method claim 2 corresponds to the apparatus disclosed in claim 8 and is rejected for the same reasons of obviousness as used above.
Regarding claim 17, Zou in view of Ma and Li teach the method of claim 2. Claim 17 also recites similar limitations to claim 9. Therefore, method claim 17 corresponds to the apparatus disclosed in claim 9 and is rejected for the same reasons of obviousness. Please see rejection of claim 9.
Claim(s) 4-5, 10-11, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zou, Ma et al, and Bolin et al (“Demoireing of Camera-Captured Screen Images Using Deep Convolutional Neural Network”), hereinafter Bolin.
Regarding claim 10, Zou in view of Ma teaches the apparatus of claim 7. Zou does not teach wherein the processing unit is also configured to: eliminate defects in the second image to obtain an updated second image, wherein the transferring unit is also configured to: perform style transfer on the updated second image according to the third image to obtain the annotated raw image. However, Bolin teaches wherein the processing unit is also configured to: eliminate defects in the second image to obtain an updated second image,
PNG
media_image1.png
335
525
media_image1.png
Greyscale
“In this paper, we propose an approach of deep convolutional neural network for demoireing screen photos. The proposed DCNN consists of a coarse-scale network and a fine-scale network. In the coarse-scale network, the input image is first downsampled and then processed by stacked residual blocks to remove the moire artifacts” – Abstract, and Fig 2.
NOTE: Bolin discloses an algorithm that takes an input image of screen-captured content of a display and removes defects such as moireing to generate an updated image. After the combination, the algorithm to eliminate defects such as moireing as taught by Bolin can be used to edit the second image obtained from the processing unit of Ma. This modification is then added to Zou’s system for obtaining a first image of a captured annotated target image. This will allow Zou’s system to capture the annotated target image on a display, generate a second image by transforming the coordinate system to match the annotated target image and generating an updated second image with removed noise to then be used for the style transferring process.
wherein the transferring unit is also configured to: perform style transfer on the updated second image according to the third image to obtain the annotated raw image.
“To further improve the realism of the final synthesized image, reduce the style domain difference between the synthesized image and the real-world image, and ensure that the style of the synthesized image and the real-world image are as close as possible, the image style transfer process is as shown in Figure 3. In step S104, based on the acquired real-world image, the generated rendered image undergoes style reshaping to obtain the synthesized image, and the determined annotation information is used as the annotation information for the synthesized image.” – Zou, Par 108
NOTE: After the combination, Zou’s style transferring model can be modified to use the generated updated second image from Bolin which has the removed defects. Then the updated second image can be used as input along with the third image to generate the annotated raw image as described in the rejection of claim 7.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Zou by incorporating the teachings of Bolin to eliminate the defects in the second image to obtain the updated second image and to perform style transfer on the updated second image according to the third image to obtain an annotated raw image. One would be motivated to make this combination because it is a common step to remove defects in images to create a clear/clean image. Using the updated second image will allow the style transfer model to create a realistic images free from any defects which will be beneficial for training machine learning models.
Regarding claim 4, the claim recites similar limitations to claim 10. Therefore, method claim 4 corresponds to the apparatus disclosed in claim 10 and is rejected for the same reasons of obviousness as used above.
Regarding claim 19, Zou in view of Ma the claim recites similar limitations to claim 10. Therefore, method claim 19 corresponds to the apparatus disclosed in claim 10 and is rejected for the same reasons of obviousness as used above.
Regarding claim 11, Zou in view of Ma and Bolin teach the apparatus of claim 10. Zou does not teach wherein the defects comprise at least one of followings: lens distortion, vignetting, noise, moire and/or water ripples caused by a displayer. However, Bolin further teaches wherein the defects comprise at least one of followings: lens distortion, vignetting, noise, moire and/or water ripples caused by a displayer.
“In this paper, we propose an approach of deep convolutional neural network for demoireing screen photos. The proposed DCNN consists of a coarse-scale network and a fine-scale network. In the coarse-scale network, the input image is first downsampled and then processed by stacked residual blocks to remove the moire artifacts” – Abstract, and Fig 2. See rejection of claim 10
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Zou by incorporating the teachings of Bolin to eliminate the defects such as more caused by a display. One would be motivated to make this combination because it is a common step to remove defects in images to create a clear/clean image. Using the updated second image will allow the style transfer model to create a realistic images free from any defects which will be beneficial for training machine learning models.
Regarding claim 5, the claim recites similar limitations to claim 11. Therefore, method claim 5 corresponds to the apparatus disclosed in claim 11 and is rejected for the same reasons of obviousness as used above.
Claim(s) 6 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zou, Ma et al, and Souche et al (US 20180012110 A1), hereinafter Souche.
Regarding claim 12, Zou in view of Ma teaches the apparatus of claim 7. Zou does not teach wherein the processing unit is also configured to: crop the second image to obtain an image containing only the target image; and map an annotation box in the target image to the cropped second image to obtain an annotated second image, wherein the transferring unit is also configured to: wherein the processing unit is also configured to: crop the second image to obtain an image containing only the target image;
“The image extraction convolutional neural network determines a location and a bounding box to extract the image of the object from the target image, and crop the target image around the bounding box to generate the extracted image of the object.” – Par 54, Lines 16-20
NOTE: Souche discloses an image extraction convolutional neural network that determines a region of interest marked by a bounding box and crops everything outside which leaves an image of only the region of interest. This functionally corresponds to cropping the second image so that only the target image remains
and map an annotation box in the target image to the cropped second image to obtain an annotated second image,
“the image extraction convolutional neural network determines a location and a size of a bounding box for each object in the target image to extract an image of each object from the target image.” – Par 54, Lines 22-25
NOTE: Souche discloses that location and size information of a bounding box is determined for an object in the target image to extract the image of an object from. This functionally corresponds to determining an annotation box of the target image that should remain after cropping the second image.
wherein the transferring unit is also configured to: perform style transfer on the annotated second image according to the third image to obtain the annotated raw image.
To further improve the realism of the final synthesized image, reduce the style domain difference between the synthesized image and the real-world image, and ensure that the style of the synthesized image and the real-world image are as close as possible, the image style transfer process is as shown in Figure 3. In step S104, based on the acquired real-world image, the generated rendered image undergoes style reshaping to obtain the synthesized image, and the determined annotation information is used as the annotation information for the synthesized image.” – Zou, Par 108
NOTE: After the combination, the annotated second image generated by the image extraction convolutional neural network taught by Souche can then be used as input into Zou’s style transfer model along with the third image to create an annotated raw image as described in the rejection of claim 7.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Zou by incorporating the teachings of Souche to crop the second image to only contain the target image and map an annotation box in the target image to obtain an annotated second image. One would be motivated to make this combination in order to preserve accurate annotations of the target image when performing style transfer. Cropping the image to only contain the target image will also prevent unwanted areas from showing up on the new generated annotated raw images.
Regarding claim 6, the claim recites similar limitations to claim 12. Therefore, method claim 6 corresponds to the apparatus disclosed in claim 12 and is rejected for the same reasons of obviousness as used above.
Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zou, Ma, Li and Bolin.
Regarding claim 18, Zou in view of Ma and Li teaches the method of claim 2. Zou does not teach wherein the processing unit is also configured to: eliminate defects in the second image to obtain an updated second image, wherein the transferring unit is also configured to: perform style transfer on the updated second image according to the third image to obtain the annotated raw image. However, Bolin teaches wherein the processing unit is also configured to: eliminate defects in the second image to obtain an updated second image,
“In this paper, we propose an approach of deep convolutional neural network for demoireing screen photos. The proposed DCNN consists of a coarse-scale network and a fine-scale network. In the coarse-scale network, the input image is first downsampled and then processed by stacked residual blocks to remove the moire artifacts” – Abstract, and Fig 2.
NOTE: Bolin discloses an algorithm that takes an input image of screen-captured content of a display and removes defects such as moireing to generate an updated image. After the combination, the algorithm to eliminate defects such as moireing as taught by Bolin can be used to edit the second image obtained from the processing unit of Ma. This modification is then added to Zou’s system for obtaining a first image of a captured annotated target image. This will allow Zou’s system to capture the annotated target image on a display, generate a second image by transforming the coordinate system to match the annotated target image and generating an updated second image with removed noise to then be used for the style transferring process.
wherein the transferring unit is also configured to: perform style transfer on the updated second image according to the third image to obtain the annotated raw image.
“To further improve the realism of the final synthesized image, reduce the style domain difference between the synthesized image and the real-world image, and ensure that the style of the synthesized image and the real-world image are as close as possible, the image style transfer process is as shown in Figure 3. In step S104, based on the acquired real-world image, the generated rendered image undergoes style reshaping to obtain the synthesized image, and the determined annotation information is used as the annotation information for the synthesized image.” – Zou, Par 108
NOTE: After the combination, Zou’s style transferring model can be modified to use the generated updated second image from Bolin which has the removed defects. Then the updated second image can be used as input along with the third image to generate the annotated raw image as described in the rejection of claim 7.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Zou by incorporating the teachings of Bolin to eliminate the defects in the second image to obtain the updated second image and to perform style transfer on the updated second image according to the third image to obtain an annotated raw image. One would be motivated to make this combination because it is a common step to remove defects in images to create a clear/clean image. Using the updated second image will allow the style transfer model to create a realistic images free from any defects which will be beneficial for training machine learning models.
Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zou, Ma, Li and Souche.
Regarding claim 20, Zou in view of Ma and Li teaches the method of claim 2. Zou does not teach wherein the processing unit is also configured to: crop the second image to obtain an image containing only the target image; and map an annotation box in the target image to the cropped second image to obtain an annotated second image, wherein the transferring unit is also configured to: perform style transfer on the annotated second image according to the third image to obtain the annotated raw image. However, Souche teaches wherein the processing unit is also configured to: crop the second image to obtain an image containing only the target image;
“The image extraction convolutional neural network determines a location and a bounding box to extract the image of the object from the target image, and crop the target image around the bounding box to generate the extracted image of the object.” – Par 54, Lines 16-20
NOTE: Souche discloses an image extraction convolutional neural network that determines a region of interest marked by a bounding box and crops everything outside which leaves an image of only the region of interest. This functionally corresponds to cropping the second image so that only the target image remains
and map an annotation box in the target image to the cropped second image to obtain an annotated second image,
“the image extraction convolutional neural network determines a location and a size of a bounding box for each object in the target image to extract an image of each object from the target image.” – Par 54, Lines 22-25
NOTE: Souche discloses that location and size information of a bounding box is determined for an object in the target image to extract the image of an object from. This functionally corresponds to determining an annotation box of the target image that should remain after cropping the second image.
wherein the transferring unit is also configured to: perform style transfer on the annotated second image according to the third image to obtain the annotated raw image.
To further improve the realism of the final synthesized image, reduce the style domain difference between the synthesized image and the real-world image, and ensure that the style of the synthesized image and the real-world image are as close as possible, the image style transfer process is as shown in Figure 3. In step S104, based on the acquired real-world image, the generated rendered image undergoes style reshaping to obtain the synthesized image, and the determined annotation information is used as the annotation information for the synthesized image.” – Zou, Par 108
NOTE: After the combination, the annotated second image generated by the image extraction convolutional neural network taught by Souche can then be used as input into Zou’s style transfer model along with the third image to create an annotated raw image as described in the rejection of claim 7.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Zou by incorporating the teachings of Souche to crop the second image to only contain the target image and map an annotation box in the target image to obtain an annotated second image. One would be motivated to make this combination in order to preserve accurate annotations of the target image when performing style transfer. Cropping the image to only contain the target image will also prevent unwanted areas from showing up on the new generated annotated raw images.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID V. NGUYEN whose telephone number is (571)272-6111. The examiner can normally be reached M-F 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Y Poon can be reached at 571-270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID VAN NGUYEN/Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights