DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 11/18/2024 and 07/10/2026 are being considered by the examiner.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
Claim 19, recites limitations that use words like “means” (or “step”) or similar terms with functional language and do invoke 35 U.S.C. 112(f):
Claim 19; recites the limitation, “obtaining, by a device, first image data…..” [Line 2].
Claim 19; recites the limitation, “obtaining, by the device, second image data…..” [Line 4].
Claim 19; recites the limitation, “identifying, by the device and based on the second image data, a region…...” [Line 7].
Claim 19; recites the limitation, “generating, by the device, fill-in image data……,” [Line 9].
Claim 19; recites the limitation, “generating, by the device, output image data……,” [Line 11].
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
After a careful analysis, as disclosed above, and a careful review of the specification the following limitations in claim 19;
(i) “device” (Fig. 1, #102 called a device. Paragraph [0036 and 0042]-FIG. 1 is a block diagram of an example of a system 100 including a device 102 operable to eliminate reflected artifacts in an image, in accordance with one or more aspects of the present disclosure. The device 102 includes, or is coupled to, a memory 106, one or more processors 108 (collectively referred to herein as the “processor 108”), a first camera 110 (e.g., an image sensor), a second camera 112, an input device 114, a display device 116, a speaker 117, and a modem 118. The memory 106 may include one or more memory devices, such as a single memory device or multiple different memory devices (of the same type or of different types). The memory 106 is configured to store instructions 109, size/field of view (FOV) data 134, and optionally, a segmentation mask 144. The input device 114 may include a keypad, a touchscreen, a microphone, a camera, or another user input device. The input data 115 represents the user input provided to the device 102. The device, illustrated in Fig. 1 as block #102, thus has sufficient structure or material, wherein is a device with a memory, a processor, a first camera, a second camera, a display, a speaker, a modem, and a touchscreen.)
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 6-7, and 11-15, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over NAKAGAWA (US 20220253993 A1), hereinafter referenced as NAKAGAWA, in view of CHANG et al. (US 20150363920 A1), hereinafter referenced as CHANG.
Regarding claim 1, NAKAGAWA explicitly teaches a device comprising (Fig. 1, #100 called smart phone. Paragraph [0022]-NAKAGAWA discloses the CPU 101 controls the operations of the functional blocks by loading programs (the OS, applications, and the like) stored in a non-volatile memory 103 into a memory 102 and executing them to realize the various functions of the smart phone 100.):
a memory (Fig. 1, #102 called memory. Paragraph [0022-0024]) configured to store first image data (Fig. 1. Paragraph [0023-0024]-NAKAGAWA discloses the memory 102 is RAM, for example, and is used as a main storage apparatus, a working memory, a buffer memory, a video memory, and/or the like. The non-volatile memory 103 is an electrically rewritable ROM, for example. The non-volatile memory 103 stores programs for the CPU 101 to execute, various setting values, system data such as graphical user interface (GUI) data, and user data, such as data of still images and moving images captured by the user, applications, and the like.) representing a first image captured by a first camera facing a first direction (Fig. 2B, illustrates a first camera called outward-facing camera #107 (wherein outward-facing is a first direction). Paragraph [0025]-NAKAGAWA discloses an image processing circuit 104 is controlled by the CPU 101 to apply various types of image processing to the image data obtained via image capture by an outward-facing camera 107.); and
one or more processors (Fig. 1, #101 called CPU and #104 called image processing circuit. Paragraph [0024]-NAKAGAWA discloses the non-volatile memory 103 is an electrically rewritable ROM, for example. The non-volatile memory 103 stores programs for the CPU 101 to execute, various setting values, system data such as graphical user interface (GUI) data, and user data, such as data of still images and moving images captured by the user, applications, and the like. Further in paragraph [0025]-NAKAGAWA discloses the image processing circuit 104 may be configured to realize a specific function via a programmable processor such as a digital signal processor (DSP) executing a program stored in the non-volatile memory 103. ), coupled to the memory (Fig. 1, #102 called memory and #101 called CPU. Paragraph [0024]-NAKAGAWA discloses the non-volatile memory 103 is an electrically rewritable ROM, for example. The non-volatile memory 103 stores programs for the CPU 101 to execute, various setting values, system data such as graphical user interface (GUI) data, and user data, such as data of still images and moving images captured by the user, applications, and the like.),
wherein the one or more processors are configured to (Fig. 1. Paragraph [0024]-NAKAGAWA discloses the non-volatile memory 103 is an electrically rewritable ROM, for example. The non-volatile memory 103 stores programs for the CPU 101 to execute, various setting values, system data such as graphical user interface (GUI) data, and user data, such as data of still images and moving images captured by the user, applications, and the like.):
obtain second image data representing a second image captured by a second camera facing a second direction that is opposite to the first direction (Fig. 2A-2B. Paragraph [0025]-NAKAGAWA discloses an image processing circuit 104 is controlled by the CPU 101 to apply various types of image processing to the image data obtained via image capture by an outward-facing camera 107, a standard inward-facing camera 109, and a wide-angle inward-facing camera 110 (wherein the inward-facing camera is a direction opposite of the outward facing camera).);
identify, based on the second image data, a region in the first image that includes one or more reflected objects (Fig. 4A. Paragraph [0043]-NAKAGAWA discloses using the image of the person capturing the image captured by the wide-angle inward-facing camera 110, reflections in the image of the subject captured by the outward-facing camera 107 are reduced. Further in paragraph [0061]-NAKAGAWA discloses the CPU 101 instructs the image processing circuit 104 to detect if there is a reflection of the user based on the image obtained by the inward-facing camera 110 and the image obtained by the outward-facing camera 107. The image processing circuit 104 uses known technology such as pattern matching to detect if there is a reflection of the user and notifies the CPU 101 of the result.);
NAKAGAWA fails to explicitly teach generate fill-in image data based on the first image data, the fill-in image data representing a fill-in image; and generate output image data, based on the first image data and the fill-in image data, that corresponds to the first image in which the region is replaced with the fill-in image.
However, CHANG explicitly teaches generate fill-in image data based on the first image data (Fig. 1. Paragraph [0031]-CHANG discloses an image inpainting algorithm may be further adopted to remove the reflection recognized in the foregoing embodiment. In detail, after the reflection determining module 148 determines which object is the reflection in the step S210 in aforesaid embodiment, a reflection processing module (not shown) stored in the storage unit 14 may be further executed by the processing unit 16 to remove the reflection from the image and inpaints a removed region of the image with other images by means of image inpainting (wherein inpainting is generating fill-in image data).), the fill-in image data representing a fill-in image (Fig. 1. Paragraph [0032]-CHANG discloses through the technique of image inpainting, the removed region of the image can be recovered by other images obtained by the image retrieving unit 12 (wherein the other images are fill-in images).); and
generate output image data, based on the first image data and the fill-in image data, that corresponds to the first image in which the region is replaced with the fill-in image (Fig. 1. Paragraph [0036]-CHANG discloses the recognized reflection can be further removed and the removed region can be inpainted with other images or neighboring regions through image inpainting. As a result, an image without reflection can be obtained and a better image quality is achieved. (wherein the input image is the first image, the region that is inpainted is the fill-in image data, and the output image data is the image without reflection).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of NAKAGAWA of a device comprising: a memory configured to store first image data representing a first image captured by a first camera facing a first direction; and one or more processors, coupled to the memory, wherein the one or more processors are configured to: obtain second image data representing a second image captured by a second camera facing a second direction that is opposite to the first direction; identify, based on the second image data, a region in the first image that includes one or more reflected objects; with the teachings of CHANG of generate fill-in image data based on the first image data, the fill-in image data representing a fill-in image; and generate output image data, based on the first image data and the fill-in image data, that corresponds to the first image in which the region is replaced with the fill-in image.
Wherein having NAKAGAWA’s method/device of image reflection removal having generate fill-in image data based on the first image data, the fill-in image data representing a fill-in image; and generate output image data, based on the first image data and the fill-in image data, that corresponds to the first image in which the region is replaced with the fill-in image.
The motivation behind the modification would have been to obtain method/device capable of efficiently and accurately removing reflections from images. Since both NAKAGAWA and CHANG relate to removing reflections from images, wherein NAKAGAWA limitations relating to capturing scenes and image capture method on an image processing apparatus and an image processing method able to reduce a reflection in a captured image are reduced, while CHANG the nearby object is recognized as a reflection and can be removed from the image so as to improve the image quality. Please see NAKAGAWA (US 20220253993 A1), Paragraph [0005], and CHANG et al. (US 20150363920 A1), Paragraph [0010].
Regarding claim 6, NAKAGAWA in view of CHANG explicitly teach the device of claim 1,
NAKAGAWA further explicitly teaches wherein the one or more processors are configured to (Fig. 1. Paragraph [0024]-NAKAGAWA discloses the non-volatile memory 103 is an electrically rewritable ROM, for example. The non-volatile memory 103 stores programs for the CPU 101 to execute, various setting values, system data such as graphical user interface (GUI) data, and user data, such as data of still images and moving images captured by the user, applications, and the like.):
perform one or more object recognition operations based on the first image data and the second image data (Fig. 6B, S611 called compare face detected in outward-facing camera image and face detected in wide-angle inward-facing camera image. Paragraph [0030]-NAKAGAWA discloses the detection processing includes processing for detecting a feature area (for example, a face area or a human body area) or movement thereof, processing for recognizing a person, and the like. Further in paragraph [0083]-NAKAGAWA discloses in step S611, the CPU 101 instructs the image processing circuit 104 to calculate the degree of match between the face detected in the video captured by the outward-facing camera 107 and the face detected in the video captured by the wide-angle inward-facing camera 110, the degree of match indicating how much these two match (wherein a face is an object).); and
identify a common object that is included in the first image and the second image based on the one or more object recognition operations (Fig. 6B, S611 called compare face detected in outward-facing camera image and face detected in wide-angle inward-facing camera image. Paragraph [0083]-NAKAGAWA discloses in step S611, the CPU 101 instructs the image processing circuit 104 to calculate the degree of match between the face detected in the video captured by the outward-facing camera 107 and the face detected in the video captured by the wide-angle inward-facing camera 110, the degree of match indicating how much these two match. The image processing circuit 104, from the face area detected in both videos, converts the position, shape, and size of the eyes, nose, mouth, and the like, the overall shape of the face, bumps and recesses in the face, and other face features in numerical values. Also, the image processing circuit 104 compares the numerical values relating the face features and calculates the degree of match between the faces (wherein a face is an object and the matched faces are a common object).),
wherein the one or more reflected objects include the common object (Fig. 6B, S611 called compare face detected in outward-facing camera image and face detected in wide-angle inward-facing camera image. Paragraph [0083]-NAKAGAWA discloses in step S611, the CPU 101 instructs the image processing circuit 104 to calculate the degree of match between the face detected in the video captured by the outward-facing camera 107 and the face detected in the video captured by the wide-angle inward-facing camera 110, the degree of match indicating how much these two match (wherein the reflect objects are the faces detected in the images from the outward-facing camera).).
Regarding claim 7, NAKAGAWA in view of CHANG explicitly teach the device of claim 1,
NAKAGAWA further explicitly teaches wherein the one or more processors are configured to (Fig. 1. Paragraph [0024]-NAKAGAWA discloses the non-volatile memory 103 is an electrically rewritable ROM, for example. The non-volatile memory 103 stores programs for the CPU 101 to execute, various setting values, system data such as graphical user interface (GUI) data, and user data, such as data of still images and moving images captured by the user, applications, and the like.):
perform one or more facial recognition operations based on the first image data and the second image data (Fig. 6B, S611 called compare face detected in outward-facing camera image and face detected in wide-angle inward-facing camera image. Paragraph [0030]-NAKAGAWA discloses the detection processing includes processing for detecting a feature area (for example, a face area or a human body area) or movement thereof, processing for recognizing a person, and the like. Further in paragraph [0083]-NAKAGAWA discloses in step S611, the CPU 101 instructs the image processing circuit 104 to calculate the degree of match between the face detected in the video captured by the outward-facing camera 107 and the face detected in the video captured by the wide-angle inward-facing camera 110, the degree of match indicating how much these two match.); and
identify a common face that is included in the first image and the second image based on the one or more facial recognition operations (Fig. 6B, S611 called compare face detected in outward-facing camera image and face detected in wide-angle inward-facing camera image. Paragraph [0083]-NAKAGAWA discloses in step S611, the CPU 101 instructs the image processing circuit 104 to calculate the degree of match between the face detected in the video captured by the outward-facing camera 107 and the face detected in the video captured by the wide-angle inward-facing camera 110, the degree of match indicating how much these two match. The image processing circuit 104, from the face area detected in both videos, converts the position, shape, and size of the eyes, nose, mouth, and the like, the overall shape of the face, bumps and recesses in the face, and other face features in numerical values. Also, the image processing circuit 104 compares the numerical values relating the face features and calculates the degree of match between the faces.),
wherein the one or more reflected objects include the common face (Fig. 6B, S611 called compare face detected in outward-facing camera image and face detected in wide-angle inward-facing camera image. Paragraph [0083]-NAKAGAWA discloses in step S611, the CPU 101 instructs the image processing circuit 104 to calculate the degree of match between the face detected in the video captured by the outward-facing camera 107 and the face detected in the video captured by the wide-angle inward-facing camera 110, the degree of match indicating how much these two match (wherein the reflect objects are the faces detected in the images from the outward-facing camera).).
Regarding claim 11, NAKAGAWA in view of CHANG explicitly teach the device of claim 1,
NAKAGAWA fails to explicitly teach wherein the output image data represents an output image that includes a first plurality of pixels corresponding to a remainder of the first image that does not include the region and a second plurality of pixels corresponding to the fill-in image that are included within the region of the output image.
However, CHANG explicitly teaches wherein the output image data represents an output image that includes a first plurality of pixels corresponding to a remainder of the first image that does not include the region and a second plurality of pixels corresponding to the fill-in image that are included within the region of the output image (Fig. 1. Paragraph [0032]-CHANG discloses after the reflection determining module 148 determines which object is the reflection in the step S210 in aforesaid embodiment, a reflection processing module (not shown) stored in the storage unit 14 may be further executed by the processing unit 16 to remove the reflection from the image and inpaints a removed region of the image with other images by means of image inpainting. To be specific, when the reflection is removed from the image, the removed region is emptied and may not be recovered (wherein the first plurality of pixels are regions of the image that are not inpainted and wherein the regions that are inpainted are a second plurality of pixels corresponding to the fill-in image). Further in paragraph [0036]-CHANG discloses the recognized reflection can be further removed and the removed region can be inpainted with other images or neighboring regions through image inpainting. As a result, an image without reflection can be obtained and a better image quality is achieved (wherein an image without reflection is an output image).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of NAKAGAWA of a device comprising: a memory configured to store first image data representing a first image captured by a first camera facing a first direction; and one or more processors, coupled to the memory, wherein the one or more processors are configured to: obtain second image data representing a second image captured by a second camera facing a second direction that is opposite to the first direction; identify, based on the second image data, a region in the first image that includes one or more reflected objects; with the teachings of CHANG of wherein the output image data represents an output image that includes a first plurality of pixels corresponding to a remainder of the first image that does not include the region and a second plurality of pixels corresponding to the fill-in image that are included within the region of the output image.
Wherein having NAKAGAWA’s method/device of image reflection removal having wherein the output image data represents an output image that includes a first plurality of pixels corresponding to a remainder of the first image that does not include the region and a second plurality of pixels corresponding to the fill-in image that are included within the region of the output image.
The motivation behind the modification would have been to obtain method/device capable of efficiently and accurately removing reflections from images. Since both NAKAGAWA and CHANG relate to removing reflections from images, wherein NAKAGAWA limitations relating to capturing scenes and image capture method on an image processing apparatus and an image processing method able to reduce a reflection in a captured image are reduced, while CHANG the nearby object is recognized as a reflection and can be removed from the image so as to improve the image quality. Please see NAKAGAWA (US 20220253993 A1), Paragraph [0005], and CHANG et al. (US 20150363920 A1), Paragraph [0010].
Regarding claim 12, NAKAGAWA in view of CHANG explicitly teach the device of claim 11,
NAKAGAWA fails to explicitly teach wherein the output image includes a third plurality of pixels corresponding to the fill-in image that are included in one or more locations adjacent to the region in the output image.
However, CHANG explicitly teaches wherein the output image includes a third plurality of pixels corresponding to the fill-in image that are included in one or more locations adjacent to the region in the output image (Fig. 1. Paragraph [0036]-CHANG discloses the recognized reflection can be further removed and the removed region can be inpainted with other images or neighboring regions through image inpainting (wherein the pixels inpainted with neighboring regions is a third plurality that are included in one or more locations adjacent to the region in the output image.).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of NAKAGAWA of a device comprising: a memory configured to store first image data representing a first image captured by a first camera facing a first direction; and one or more processors, coupled to the memory, wherein the one or more processors are configured to: obtain second image data representing a second image captured by a second camera facing a second direction that is opposite to the first direction; identify, based on the second image data, a region in the first image that includes one or more reflected objects; with the teachings of CHANG of wherein the output image includes a third plurality of pixels corresponding to the fill-in image that are included in one or more locations adjacent to the region in the output image.
Wherein having NAKAGAWA’s method/device of image reflection removal having wherein the output image includes a third plurality of pixels corresponding to the fill-in image that are included in one or more locations adjacent to the region in the output image.
The motivation behind the modification would have been to obtain method/device capable of efficiently and accurately removing reflections from images. Since both NAKAGAWA and CHANG relate to removing reflections from images, wherein NAKAGAWA limitations relating to capturing scenes and image capture method on an image processing apparatus and an image processing method able to reduce a reflection in a captured image are reduced, while CHANG the nearby object is recognized as a reflection and can be removed from the image so as to improve the image quality. Please see NAKAGAWA (US 20220253993 A1), Paragraph [0005], and CHANG et al. (US 20150363920 A1), Paragraph [0010].
Regarding claim 13, NAKAGAWA in view of CHANG explicitly teach the device of claim 1, further comprising:
NAKAGAWA further explicitly teaches the first camera coupled to the one or more processors and configured to generate the first image data (Fig. 1, #107 called outward-facing camera and #101 called CPU. Paragraph [0025]-NAKAGAWA discloses an image processing circuit 104 is controlled by the CPU 101 to apply various types of image processing to the image data obtained via image capture by an outward-facing camera 107.),
wherein the first camera is integrated in a back side of the device (Fig. 2B, illustrates outward-facing camera #107 on the back side of the device. Paragraph [0038]-NAKAGAWA discloses if the display surface of the display 105 is defined as the front surface of the smart phone 100, the outward-facing camera 107 is provided on the rear surface of the smart phone 100.).
Regarding claim 14, NAKAGAWA in view of CHANG explicitly teach the device of claim 13, further comprising:
NAKAGAWA further explicitly teaches the second camera coupled to the one or more processors and configured to generate the second image data (Fig. 1, #109 called inward-facing camera and #101 called CPU. Paragraph [0025]-NAKAGAWA discloses an image processing circuit 104 is controlled by the CPU 101 to apply various types of image processing to the image data obtained via image capture by an outward-facing camera 107, a standard inward-facing camera 109.),
wherein the second camera is integrated in a front side of the device (Fig. 2A, illustrates inward-facing camera #109 on the front side of the device. Paragraph [0040]-NAKAGAWA discloses the standard inward-facing camera 109 and the wide-angle inward-facing camera 110 are provided on the front surface of the smart phone 100.).
Regarding claim 15, NAKAGAWA in view of CHANG explicitly teach the device of claim 1, further comprising:
NAKAGAWA further explicitly teaches a display coupled to the one or more processors (Fig. 1, #105 called display and #101 called CPU. Paragraph [0034]-NAKAGAWA discloses a display 105 is a touch liquid crystal display, for example. The display 105 is controlled by the CPU 101 to display images (icons, windows, software keys, and the like) of the GUI of the OS or applications, images captured by a camera, and the like.) and configured to display an output image based on the output image data (Fig. 1 and 4C. Paragraph [0062]-NAKAGAWA discloses in a case where the CPU 101 is notified by the image processing circuit 104 that there is a reflection of the user, the CPU 101 instructs the image processing circuit 104 to display the video obtained by the wide-angle inward-facing camera 110 in a picture-in-picture mode, for example. In response to the instruction, the image processing circuit 104 starts the picture-in-picture display of the video obtained by the wide-angle inward-facing camera 110 (FIG. 4C).).
Regarding claim 20, NAKAGAWA explicitly teaches a non-transitory computer-readable medium storing instructions that (Fig. 1. Paragraph [0105]-NAKAGAWA discloses embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s).), when executed by one or more processors (Fig. 1, #1 called CPU. Paragraph [0105]-NAKAGAWA discloses the computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions.), cause the one or more processors to (Fig. 1, #1 called CPU. Paragraph [0105]-NAKAGAWA discloses the computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions.):
obtain first image data representing a first image captured by a first camera facing a first direction (Fig. 2B, illustrates a first camera called outward-facing camera #107 (wherein outward-facing is a first direction). Paragraph [0025]-NAKAGAWA discloses an image processing circuit 104 is controlled by the CPU 101 to apply various types of image processing to the image data obtained via image capture by an outward-facing camera 107.);
obtain second image data representing a second image captured by a second camera facing a second direction that is opposite to the first direction (Fig. 2A-2B. Paragraph [0025]-NAKAGAWA discloses an image processing circuit 104 is controlled by the CPU 101 to apply various types of image processing to the image data obtained via image capture by an outward-facing camera 107, a standard inward-facing camera 109, and a wide-angle inward-facing camera 110 (wherein the inward-facing camera is a direction opposite of the outward facing camera).);
identify, based on the second image data, a region in the first image that includes one or more reflected objects (Fig. 4A. Paragraph [0043]-NAKAGAWA discloses using the image of the person capturing the image captured by the wide-angle inward-facing camera 110, reflections in the image of the subject captured by the outward-facing camera 107 are reduced. Further in paragraph [0061]-NAKAGAWA discloses the CPU 101 instructs the image processing circuit 104 to detect if there is a reflection of the user based on the image obtained by the inward-facing camera 110 and the image obtained by the outward-facing camera 107. The image processing circuit 104 uses known technology such as pattern matching to detect if there is a reflection of the user and notifies the CPU 101 of the result.);
NAKAGAWA fails to explicitly teach generate fill-in image data based on the first image data, the fill-in image data representing a fill-in image; and generate output image data, based on the first image data and the fill-in image data, that corresponds to the first image in which the region is replaced with the fill-in image.
However, CHANG explicitly teaches generate fill-in image data based on the first image data (Fig. 1. Paragraph [0031]-CHANG discloses it is noted that, in another embodiment, an image inpainting algorithm may be further adopted to remove the reflection recognized in the foregoing embodiment. In detail, after the reflection determining module 148 determines which object is the reflection in the step S210 in aforesaid embodiment, a reflection processing module (not shown) stored in the storage unit 14 may be further executed by the processing unit 16 to remove the reflection from the image and inpaints a removed region of the image with other images by means of image inpainting (wherein inpainting is generating fill-in image data).), the fill-in image data representing a fill-in image (Fig. 1. Paragraph [0032]-CHANG discloses through the technique of image inpainting, the removed region of the image can be recovered by other images obtained by the image retrieving unit 12 (wherein the other images are fill-in images).); and
generate output image data, based on the first image data and the fill-in image data, that corresponds to the first image in which the region is replaced with the fill-in image (Fig. 1. Paragraph [0036]-CHANG discloses the recognized reflection can be further removed and the removed region can be inpainted with other images or neighboring regions through image inpainting. As a result, an image without reflection can be obtained and a better image quality is achieved. (wherein the input image is the first image, the region that is inpainted is the fill-in image data, and the output image data is the image without reflection).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of NAKAGAWA of a non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: obtain first image data representing a first image captured by a first camera facing a first direction; obtain second image data representing a second image captured by a second camera facing a second direction that is opposite to the first direction; identify, based on the second image data, a region in the first image that includes one or more reflected objects; with the teachings of CHANG of generate fill-in image data based on the first image data, the fill-in image data representing a fill-in image; and generate output image data, based on the first image data and the fill-in image data, that corresponds to the first image in which the region is replaced with the fill-in image.
Wherein having NAKAGAWA’s method/device of image reflection removal having generate fill-in image data based on the first image data, the fill-in image data representing a fill-in image; and generate output image data, based on the first image data and the fill-in image data, that corresponds to the first image in which the region is replaced with the fill-in image.
The motivation behind the modification would have been to obtain method/device capable of efficiently and accurately removing reflections from images. Since both NAKAGAWA and CHANG relate to removing reflections from images, wherein NAKAGAWA limitations relating to capturing scenes and image capture method on an image processing apparatus and an image processing method able to reduce a reflection in a captured image are reduced, while CHANG the nearby object is recognized as a reflection and can be removed from the image so as to improve the image quality. Please see NAKAGAWA (US 20220253993 A1), Paragraph [0005], and CHANG et al. (US 20150363920 A1), Paragraph [0010].
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over NAKAGAWA (US 20220253993 A1), hereinafter referenced as NAKAGAWA, in view of CHANG et al. (US 20150363920 A1), hereinafter referenced as CHANG, and further in view of NA et al. (US 20130265311 A1), hereinafter referenced as NA.
Regarding claim 2, NAKAGAWA in view of CHANG explicitly teach the device of claim 1,
NAKAGAWA further explicitly teaches wherein the one or more processors are configured to (Fig. 1. Paragraph [0024]-NAKAGAWA discloses the non-volatile memory 103 is an electrically rewritable ROM, for example. The non-volatile memory 103 stores programs for the CPU 101 to execute, various setting values, system data such as graphical user interface (GUI) data, and user data, such as data of still images and moving images captured by the user, applications, and the like.),
NAKAGAWA in view of CHANG fail to explicitly teach prior to identification of the region in the first image, perform one or more resizing operations on the second image data based on size information associated with the first image data.
However, NA explicitly teaches prior to identification of the region in the first image (Fig. 6. [0057]-NA discloses subsequently, the image processing device sets an interest area 615 in the resized image 600 based on the selected area 505 of the first image 500 (wherein a selected area is an identified region).), perform one or more resizing operations on the second image data based on size information associated with the first image data (Fig. 6. Paragraph [0056]-NA discloses upon receiving a full-size or high-resolution second image 510, the second image processor 140 resizes the high-resolution second image 510 to be equal in size to the low-resolution first image 500.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of NAKAGAWA in view of CHANG of a device comprising: a memory configured to store first image data representing a first image captured by a first camera facing a first direction; and one or more processors, coupled to the memory, wherein the one or more processors are configured to: obtain second image data representing a second image captured by a second camera facing a second direction that is opposite to the first direction; identify, based on the second image data, a region in the first image that includes one or more reflected objects; with the teachings of NA of prior to identification of the region in the first image, perform one or more resizing operations on the second image data based on size information associated with the first image data.
Wherein having NAKAGAWA’s method/device of image reflection removal having prior to identification of the region in the first image, perform one or more resizing operations on the second image data based on size information associated with the first image data.
The motivation behind the modification would have been to obtain method/device capable of efficiently and accurately removing reflections from images. Since both NAKAGAWA and NA relate to improving image quality in regions of an image, wherein NAKAGAWA limitations relating to capturing scenes and image capture method on an image processing apparatus and an image processing method able to reduce a reflection in a captured image are reduced, while NA an image processing device and method is provided, for simultaneously displaying a preview image and an enlarged image of a selected area without degradation of the image quality. Please see NAKAGAWA (US 20220253993 A1), Paragraph [0005], and NA et al. (US 20130265311 A1), Paragraph [0010].
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over NAKAGAWA (US 20220253993 A1), hereinafter referenced as NAKAGAWA, in view of CHANG et al. (US 20150363920 A1), hereinafter referenced as CHANG, and further in view of HA et al. (US 20220294986 A1), hereinafter referenced as HA.
Regarding claim 3, NAKAGAWA in view of CHANG explicitly teach the device of claim 1,
NAKAGAWA further explicitly teaches wherein the one or more processors are configured to (Fig. 1. Paragraph [0024]-NAKAGAWA discloses the non-volatile memory 103 is an electrically rewritable ROM, for example. The non-volatile memory 103 stores programs for the CPU 101 to execute, various setting values, system data such as graphical user interface (GUI) data, and user data, such as data of still images and moving images captured by the user, applications, and the like.),
NAKAGAWA in view of CHANG fail to explicitly teach prior to identification of the region in the first image, perform one or more field of view (FOV) correction operations on the second image data based on FOV information associated with the first camera and the second camera.
However, HA explicitly teaches prior to identification of the region in the first image (Fig. 1-2. Paragraph [0079]-HA discloses the processor 210 may determine and crop a first area in at least one initial frame obtained after switching to the second camera module 120, based on the calibration data, and track the first object from the second image data in the following frames to determine a first area including the first object.), perform one or more field of view (FOV) correction operations on the second image data based on FOV information associated with the first camera and the second camera (Fig. 2. Paragraph [0079]-HA discloses when an input of image data is switched from the first camera module 110 to the second camera module 120, the processor 210 may determine a first area corresponding to the first FOV of the first camera module 110 from the second image data of the second camera module 120, and crop the first area from the second image data, thereby generating an output image.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of NAKAGAWA in view of CHANG of a device comprising: a memory configured to store first image data representing a first image captured by a first camera facing a first direction; and one or more processors, coupled to the memory, wherein the one or more processors are configured to: obtain second image data representing a second image captured by a second camera facing a second direction that is opposite to the first direction; identify, based on the second image data, a region in the first image that includes one or more reflected objects; with the teachings of HA of prior to identification of the region in the first image, perform one or more field of view (FOV) correction operations on the second image data based on FOV information associated with the first camera and the second camera.
Wherein having NAKAGAWA’s method/device of image reflection removal having prior to identification of the region in the first image, perform one or more field of view (FOV) correction operations on the second image data based on FOV information associated with the first camera and the second camera.
The motivation behind the modification would have been to obtain method/device capable of efficiently and accurately removing reflections from images. Since both NAKAGAWA and HA relate to analyzing regions of images with the employment of multiple cameras, wherein NAKAGAWA limitations relating to capturing scenes and image capture method on an image processing apparatus and an image processing method able to reduce a reflection in a captured image are reduced, while HA is directed to provide an electronic device using two cameras and configured to perform quick and smooth switching between the cameras by automatically performing switching of the cameras based on an object's movement. Please see NAKAGAWA (US 20220253993 A1), Paragraph [0005], and HA et al. (US 20220294986 A1), Paragraph [0022].
Claims 4-5 are rejected under 35 U.S.C. 103 as being unpatentable over NAKAGAWA (US 20220253993 A1), hereinafter referenced as NAKAGAWA, in view of CHANG et al. (US 20150363920 A1), hereinafter referenced as CHANG, and further in view of KIM et al. (US 20200167943 A1), hereinafter referenced as KIM, and further in view of TSAI et al. (US 11276177 B1), hereinafter referenced as TSAI.
Regarding claim 4, NAKAGAWA in view of CHANG explicitly teach the device of claim 1,
NAKAGAWA further explicitly teaches wherein the one or more processors are configured to (Fig. 1. Paragraph [0024]-NAKAGAWA discloses the non-volatile memory 103 is an electrically rewritable ROM, for example. The non-volatile memory 103 stores programs for the CPU 101 to execute, various setting values, system data such as graphical user interface (GUI) data, and user data, such as data of still images and moving images captured by the user, applications, and the like.):
NAKAGAWA in view of CHANG fail to explicitly teach generate a segmentation mask based on the second image; and.
However, KIM explicitly teaches generate a segmentation mask based on the second image (Fig. 3. Paragraph [0043]-KIM discloses a refined segmentation mask is generated for the input image based on segmentation masks for each plane object in the number of plane objects. In an embodiment, each plane object detected by the plane detection network is associated with a corresponding segmentation mask for that plane object. The segmentation mask can be a binary mask that associates each pixel in a portion of the image or a portion of the region of interest as being either a part of the plane object or a part of the background of the image (wherein the input image is the second image).); and
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of NAKAGAWA in view of CHANG of a device comprising: a memory configured to store first image data representing a first image captured by a first camera facing a first direction; and one or more processors, coupled to the memory, wherein the one or more processors are configured to: obtain second image data representing a second image captured by a second camera facing a second direction that is opposite to the first direction; identify, based on the second image data, a region in the first image that includes one or more reflected objects; with the teachings of KIM of generate a segmentation mask based on the second image; and.
Wherein having NAKAGAWA’s method/device of image reflection removal having generate a segmentation mask based on the second image; and.
The motivation behind the modification would have been to obtain method/device capable of efficiently and accurately removing reflections from images. Since both NAKAGAWA and KIM relate to analyzing specific regions in an image to identify subjects, wherein NAKAGAWA limitations relating to capturing scenes and image capture method on an image processing apparatus and an image processing method able to reduce a reflection in a captured image are reduced, while KIM states approaches suffer from some limitations such as: (1) missing small surfaces; (2) requiring a maximum number of planes in a single image, a priori; and (3) poor generalization across domains. Therefore, there is a need for addressing these issues and/or other issues associated with the prior art. Please see NAKAGAWA (US 20220253993 A1), Paragraph [0005], and KIM et al. (US 20200167943 A1), Paragraph [0003].
NAKAGAWA in view of CHANG and further in view of KIM fail to explicitly teach perform a comparison of the segmentation mask and the first image, wherein the region in the first image is identified based on the segmentation mask.
However, TSAI explicitly teaches perform a comparison of the segmentation mask and the first image (Fig. 2A-2B, illustrate comparing the segmentation mask with the first image. Col. 13, Lines [9-15]-TSAI discloses while the segmentation mask 242 accurately captures the cup and the user, the segmentation mask 252 captures the user but fails to properly capture the cup. As shown, the segmentation masks 242 and 252 produced from the two images 240 and 250 are inconsistent, which results in inconsistent segmentation results and can impact image processing tasks that rely on the segmentation results.),
wherein the region in the first image is identified based on the segmentation mask (Fig. 3A-3B. Col. 16, Lines [25-28]-TSAI discloses the white portions of the segmentation mask 330 can correspond to pixels classified as foreground pixels and the black portions of the segmentation mask 330 can correspond to pixels classified as background pixels (wherein the foreground pixels are the region).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of NAKAGAWA in view of CHANG and further in view of KIM of a device comprising: a memory configured to store first image data representing a first image captured by a first camera facing a first direction; and one or more processors, coupled to the memory, wherein the one or more processors are configured to: obtain second image data representing a second image captured by a second camera facing a second direction that is opposite to the first direction; identify, based on the second image data, a region in the first image that includes one or more reflected objects; with the teachings of TSAI of perform a comparison of the segmentation mask and the first image, wherein the region in the first image is identified based on the segmentation mask.
Wherein having NAKAGAWA’s method/device of image reflection removal having perform a comparison of the segmentation mask and the first image, wherein the region in the first image is identified based on the segmentation mask.
The motivation behind the modification would have been to obtain method/device capable of efficiently and accurately removing reflections from images. Since both NAKAGAWA and TSAI relate to analyzing regions in images captured by cameras with different FOVs, wherein NAKAGAWA limitations relating to capturing scenes and image capture method on an image processing apparatus and an image processing method able to reduce a reflection in a captured image are reduced, while TSAI the technologies described herein can leverage images with different field-of-views (FOVs) and/or captured using different types of lenses to improve and refine segmentation results. Please see NAKAGAWA (US 20220253993 A1), Paragraph [0005], and TSAI et al. (US 11276177 B1), Col. 6, Lines [51-62].
Regarding claim 5, NAKAGAWA in view of CHANG and further in view of KIM and further in view of TSAI explicitly teach the device of claim 4,
NAKAGAWA further explicitly teaches wherein the one or more processors are configured to (Fig. 1. Paragraph [0024]-NAKAGAWA discloses the non-volatile memory 103 is an electrically rewritable ROM, for example. The non-volatile memory 103 stores programs for the CPU 101 to execute, various setting values, system data such as graphical user interface (GUI) data, and user data, such as data of still images and moving images captured by the user, applications, and the like.):
NAKAGAWA in view of CHANG fail to explicitly teach perform one or more resizing operations on the segmentation mask based on size information associated with the first image data.
However, KIM explicitly teaches perform one or more resizing operations on the segmentation mask based on size information associated with the first image data (Fig. 6. Paragraph [0136]-KIM discloses the m×m segmentation mask 640 for each plane object can be resized to match the size of the original input image 202.);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of NAKAGAWA in view of CHANG of a device comprising: a memory configured to store first image data representing a first image captured by a first camera facing a first direction; and one or more processors, coupled to the memory, wherein the one or more processors are configured to: obtain second image data representing a second image captured by a second camera facing a second direction that is opposite to the first direction; identify, based on the second image data, a region in the first image that includes one or more reflected objects; with the teachings of KIM of perform one or more resizing operations on the segmentation mask based on size information associated with the first image data.
Wherein having NAKAGAWA’s method/device of image reflection removal perform one or more resizing operations on the segmentation mask based on size information associated with the first image data.
The motivation behind the modification would have been to obtain method/device capable of efficiently and accurately removing reflections from images. Since both NAKAGAWA and KIM relate to analyzing specific regions in an image to identify subjects, wherein NAKAGAWA limitations relating to capturing scenes and image capture method on an image processing apparatus and an image processing method able to reduce a reflection in a captured image are reduced, while KIM states approaches suffer from some limitations such as: (1) missing small surfaces; (2) requiring a maximum number of planes in a single image, a priori; and (3) poor generalization across domains. Therefore, there is a need for addressing these issues and/or other issues associated with the prior art. Please see NAKAGAWA (US 20220253993 A1), Paragraph [0005], and KIM et al. (US 20200167943 A1), Paragraph [0003].
NAKAGAWA in view of CHANG fail to explicitly teach perform one or more field of view (FOV) correction operations on the segmentation mask based on FOV information associated with the first camera and the second camera; perform one or more transformation operations on the segmentation mask based on a focal length of the first camera; or a combination thereof.
However, TSAI explicitly teaches perform one or more field of view (FOV) correction operations on the segmentation mask based on FOV information associated with the first camera and the second camera (Fig. 1. Col. 6, Lines [66-67] and Col. 7, Lines [1-12]-TSAI discloses the images captured by the different cameras having the different FOVs or focal lengths can be used as inputs to generate improved and refined segmentation results. For example, in some cases, a multi-camera apparatus can use different cameras to capture a first image with a FOV (e.g., a wide-angle FOV, a narrow-angle FOV, zoomed FOV, etc.) and a second image with a different FOV. The first image can be used to generate a first segmentation mask and the second image can be used to generate a second segmentation mask. The first and second segmentation masks can be fused to generate improved segmentation results. The segmentation results can be used to render an effect on the first image, the second image, and/or a future image (wherein a field of view correction is fusing of the segmentation masks of masks taken using cameras with different FOVs).);
perform one or more transformation operations on the segmentation mask based on a focal length of the first camera (Fig. 1. Col. 7, Lines [15-25]-TSAI discloses the images captured using different FOVs or focal lengths can be used as inputs to generate improved and refined segmentation results. For example, in some cases, a single camera can capture a first image with a FOV (e.g., a wide-angle FOV, a narrow-angle FOV, zoomed FOV, etc.) and a second image with a different FOV. The first image can be used to generate a first segmentation mask and the second image can be used to generate a second segmentation mask. The first and second segmentation masks can be fused to generate improved segmentation results. The segmentation results can be used to render an effect on the first image, the second image, and/or a future image (wherein a transformation operation is fusing the segmentation masks and wherein the fusing is based on a focal length of the first camera as different focal lengths are used in the images to capture segmentation masks).); or a combination thereof.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of NAKAGAWA in view of CHANG and further in view of KIM of a device comprising: a memory configured to store first image data representing a first image captured by a first camera facing a first direction; and one or more processors, coupled to the memory, wherein the one or more processors are configured to: obtain second image data representing a second image captured by a second camera facing a second direction that is opposite to the first direction; identify, based on the second image data, a region in the first image that includes one or more reflected objects; with the teachings of TSAI of perform one or more field of view (FOV) correction operations on the segmentation mask based on FOV information associated with the first camera and the second camera; perform one or more transformation operations on the segmentation mask based on a focal length of the first camera; or a combination thereof.
Wherein having NAKAGAWA’s method/device of image reflection removal having perform one or more field of view (FOV) correction operations on the segmentation mask based on FOV information associated with the first camera and the second camera; perform one or more transformation operations on the segmentation mask based on a focal length of the first camera; or a combination thereof.
The motivation behind the modification would have been to obtain method/device capable of efficiently and accurately removing reflections from images. Since both NAKAGAWA and TSAI relate to analyzing regions in images captured by cameras with different FOVs, wherein NAKAGAWA limitations relating to capturing scenes and image capture method on an image processing apparatus and an image processing method able to reduce a reflection in a captured image are reduced, while TSAI the technologies described herein can leverage images with different field-of-views (FOVs) and/or captured using different types of lenses to improve and refine segmentation results. Please see NAKAGAWA (US 20220253993 A1), Paragraph [0005], and TSAI et al. (US 11276177 B1), Col. 6, Lines [51-62].
Claims 8-10 are rejected under 35 U.S.C. 103 as being unpatentable over NAKAGAWA (US 20220253993 A1), hereinafter referenced as NAKAGAWA, in view of CHANG et al. (US 20150363920 A1), hereinafter referenced as CHANG, and further in view of LIM et al. (US 20180293734 A1), hereinafter referenced as LIM.
Regarding claim 8, NAKAGAWA in view of CHANG explicitly teach the device of claim 1,
NAKAGAWA further explicitly teaches wherein the one or more processors are configured to (Fig. 1. Paragraph [0024]-NAKAGAWA discloses the non-volatile memory 103 is an electrically rewritable ROM, for example. The non-volatile memory 103 stores programs for the CPU 101 to execute, various setting values, system data such as graphical user interface (GUI) data, and user data, such as data of still images and moving images captured by the user, applications, and the like.)
NAKAGAWA in view of CHANG fail to explicitly teach perform on-device generation of the fill-in image data utilizing a trained artificial intelligence image generator.
However, LIM explicitly teaches perform on-device generation of the fill-in image data (Fig. 1. Paragraph [0024]-LIM discloses the generator sub-network 102 can generate the foreground image(s) 110 in an attempt to create additional images of the anomalies shown in the training images 108 without making copies of the training images 108 or without merely subtracting out the backgrounds of the training images 108. In one embodiment, the foreground images 110 include or depict only those pixels representing the anomalies. That is, no part of the background equipment may be shown (wherein fill-in image data is the foreground images created by the generator).) utilizing a trained artificial intelligence image generator (Fig. 1, #102 called generator sub-network. Paragraph [0019]-LIM discloses the neural network 102 represents a generator sub-network of a GAN, and the neural network 104 represents a discriminator sub-network of the GAN in one embodiment.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of NAKAGAWA in view of CHANG of a device comprising: a memory configured to store first image data representing a first image captured by a first camera facing a first direction; and one or more processors, coupled to the memory, wherein the one or more processors are configured to: obtain second image data representing a second image captured by a second camera facing a second direction that is opposite to the first direction; identify, based on the second image data, a region in the first image that includes one or more reflected objects; with the teachings of LIM of perform on-device generation of the fill-in image data utilizing a trained artificial intelligence image generator.
Wherein having NAKAGAWA’s method/device of image reflection removal having perform on-device generation of the fill-in image data utilizing a trained artificial intelligence image generator.
The motivation behind the modification would have been to obtain method/device capable of efficiently and accurately removing reflections from images. Since both NAKAGAWA and LIM relate to removing anomalies from images, wherein NAKAGAWA limitations relating to capturing scenes and image capture method on an image processing apparatus and an image processing method able to reduce a reflection in a captured image are reduced, while LIM the systems and methods both improve the image reconstruction and adversarial losses to ensure that the systems and methods generate plausible images for normal scenes that are aligned to the original images. Please see NAKAGAWA (US 20220253993 A1), Paragraph [0005], and LIM et al. (US 20180293734 A1), Paragraph [0015].
Regarding claim 9, NAKAGAWA in view of CHANG and further in view of LIM explicitly teach the device of claim 8,
NAKAGAWA further explicitly teaches wherein the one or more processors are configured to (Fig. 1. Paragraph [0024]-NAKAGAWA discloses the non-volatile memory 103 is an electrically rewritable ROM, for example. The non-volatile memory 103 stores programs for the CPU 101 to execute, various setting values, system data such as graphical user interface (GUI) data, and user data, such as data of still images and moving images captured by the user, applications, and the like.)
NAKAGAWA in view of CHANG fail to explicitly teach provide a portion of the first image data as input to the trained artificial intelligence image generator to generate the fill-in image data, wherein the portion of the first image data corresponds to the region in the first image.
However, LIM explicitly teaches provide a portion of the first image data (Fig. 1, #110 called foreground image and #112 called background image.) as input to the trained artificial intelligence image generator to generate the fill-in image data (Fig. 1 and 3. Paragraph [0033]-LIM discloses the consolidated image 314 combines the created anomalies 315, 316 in the foreground image 310 created by the generator sub-network 102 with the background in the background image 312 also created by the generator sub-network 102 based on the training images 308. The consolidated image 414 combines the created anomalies 415, 416 in the foreground image 410 created by the generator sub-network 102 with the background in the background image 412 also created by the generator sub-network 102 based on the training images 408. The generator sub-network 102 can create a single consolidated image from the foreground and background images, or can create multiple, different consolidated images from multiple, different foreground images and background images (wherein the generator created anomalies is fill-in image data).),
wherein the portion of the first image data corresponds to the region in the first image (Fig. 1, #108 called images, #110 called foreground images, and #112 called background images. Paragraph [0023]-LIM discloses one or more foreground images 110 are generated by the generator sub-network 102. The generator sub-network 102 can generate the foreground image(s) 110 based on the distributions and probabilities of pixel characteristics that were determined at 204. For example, in the images 108 of one or more anomalies, certain neighboring pixels can have greater probabilities of similar appearances. These pixels can represent segments of anomalies due to the similar appearances of the pixels (wherein the anomalies indicate the region).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of NAKAGAWA in view of CHANG of a device comprising: a memory configured to store first image data representing a first image captured by a first camera facing a first direction; and one or more processors, coupled to the memory, wherein the one or more processors are configured to: obtain second image data representing a second image captured by a second camera facing a second direction that is opposite to the first direction; identify, based on the second image data, a region in the first image that includes one or more reflected objects; with the teachings of LIM of provide a portion of the first image data as input to the trained artificial intelligence image generator to generate the fill-in image data, wherein the portion of the first image data corresponds to the region in the first image.
Wherein having NAKAGAWA’s method/device of image reflection removal having provide a portion of the first image data as input to the trained artificial intelligence image generator to generate the fill-in image data, wherein the portion of the first image data corresponds to the region in the first image.
The motivation behind the modification would have been to obtain method/device capable of efficiently and accurately removing reflections from images. Since both NAKAGAWA and LIM relate to removing anomalies from images, wherein NAKAGAWA limitations relating to capturing scenes and image capture method on an image processing apparatus and an image processing method able to reduce a reflection in a captured image are reduced, while LIM the systems and methods both improve the image reconstruction and adversarial losses to ensure that the systems and methods generate plausible images for normal scenes that are aligned to the original images. Please see NAKAGAWA (US 20220253993 A1), Paragraph [0005], and LIM et al. (US 20180293734 A1), Paragraph [0015].
Regarding claim 10, NAKAGAWA in view of CHANG and further in view of LIM explicitly teach the device of claim 8,
NAKAGAWA further explicitly teaches wherein the one or more processors are configured to (Fig. 1. Paragraph [0024]-NAKAGAWA discloses the non-volatile memory 103 is an electrically rewritable ROM, for example. The non-volatile memory 103 stores programs for the CPU 101 to execute, various setting values, system data such as graphical user interface (GUI) data, and user data, such as data of still images and moving images captured by the user, applications, and the like.)
NAKAGAWA in view of CHANG fail to explicitly teach provide a portion of the first image data as input to the trained artificial intelligence image generator to generate the fill-in image data, wherein the portion of the first image data corresponds to a remainder of the first image that does not include the region.
However, LIM explicitly teaches provide a portion of the first image data (Fig. 1, #110 called foreground image and #112 called background image.) as input to the trained artificial intelligence image generator to generate the fill-in image data (Fig. 1 and 3. Paragraph [0033]-LIM discloses the consolidated image 314 combines the created anomalies 315, 316 in the foreground image 310 created by the generator sub-network 102 with the background in the background image 312 also created by the generator sub-network 102 based on the training images 308. The consolidated image 414 combines the created anomalies 415, 416 in the foreground image 410 created by the generator sub-network 102 with the background in the background image 412 also created by the generator sub-network 102 based on the training images 408. The generator sub-network 102 can create a single consolidated image from the foreground and background images, or can create multiple, different consolidated images from multiple, different foreground images and background images (wherein the generator created anomalies is fill-in image data).),
wherein the portion of the first image data corresponds to a remainder of the first image that does not include the region (Fig. 1. Paragraph [0027]-LIM discloses the generator sub-network 102 can generate the background image(s) 112 to create additional images that do not include the anomalies shown in the training images 108. The background images 112 can be created without making copies of the training images 108 or without merely subtracting out the foregrounds of the training images 108. In one embodiment, the background images 112 include or depict only those pixels that do not represent the anomalies (wherein the background images without anomalies is first image data that does not include the region).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of NAKAGAWA in view of CHANG of a device comprising: a memory configured to store first image data representing a first image captured by a first camera facing a first direction; and one or more processors, coupled to the memory, wherein the one or more processors are configured to: obtain second image data representing a second image captured by a second camera facing a second direction that is opposite to the first direction; identify, based on the second image data, a region in the first image that includes one or more reflected objects; with the teachings of LIM of provide a portion of the first image data as input to the trained artificial intelligence image generator to generate the fill-in image data, wherein the portion of the first image data corresponds to a remainder of the first image that does not include the region.
Wherein having NAKAGAWA’s method/device of image reflection removal having provide a portion of the first image data as input to the trained artificial intelligence image generator to generate the fill-in image data, wherein the portion of the first image data corresponds to a remainder of the first image that does not include the region.
The motivation behind the modification would have been to obtain method/device capable of efficiently and accurately removing reflections from images. Since both NAKAGAWA and LIM relate to removing anomalies from images, wherein NAKAGAWA limitations relating to capturing scenes and image capture method on an image processing apparatus and an image processing method able to reduce a reflection in a captured image are reduced, while LIM the systems and methods both improve the image reconstruction and adversarial losses to ensure that the systems and methods generate plausible images for normal scenes that are aligned to the original images. Please see NAKAGAWA (US 20220253993 A1), Paragraph [0005], and LIM et al. (US 20180293734 A1), Paragraph [0015].
Claims 16 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over NAKAGAWA (US 20220253993 A1), hereinafter referenced as NAKAGAWA, in view of CHANG et al. (US 20150363920 A1), hereinafter referenced as CHANG, and further in view of PARK et al. (US 20130235223 A1), hereinafter referenced as PARK.
Regarding claim 16, NAKAGAWA in view of CHANG explicitly teach the device of claim 1, further comprising:
NAKAGAWA in view of CHANG fail to explicitly teach a modem coupled to the one or more processors and configured to receive the first image data, the second image data, or a combination thereof.
However, PARK explicitly teaches a modem coupled to the one or more processors (Fig. 1, #50 called modem and #20 called processor. Paragraph [0062]) and configured to receive the first image data, the second image data, or a combination thereof (Fig. 1 and 4. Paragraph [0077]-PARK discloses the processor 20 produces the composite digital video sequence 411 and stores it in the image memory 30, or provides it for real-time transmission using the wireless modem 50. Further in paragraph [0080]-PARK discloses the video multiplexer 405 forms a composite digital video sequence 411 by combining the first digital video sequence 401 and the second digital video sequence 402 and stores the composite digital video sequence 411 to the image memory 30 (wherein the video sequence includes first image data and second image data).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of NAKAGAWA in view of CHANG of a device comprising: a memory configured to store first image data representing a first image captured by a first camera facing a first direction; and one or more processors, coupled to the memory, wherein the one or more processors are configured to: obtain second image data representing a second image captured by a second camera facing a second direction that is opposite to the first direction; identify, based on the second image data, a region in the first image that includes one or more reflected objects; with the teachings of PARK of a modem coupled to the one or more processors and configured to receive the first image data, the second image data, or a combination thereof.
Wherein having NAKAGAWA’s method/device of image reflection removal having a modem coupled to the one or more processors and configured to receive the first image data, the second image data, or a combination thereof.
The motivation behind the modification would have been to obtain method/device capable of efficiently and accurately removing reflections from images that has widescale fast internet connection. Since both NAKAGAWA and PARK relate to using opposing cameras to analyze specific regions of an image, wherein NAKAGAWA limitations relating to capturing scenes and image capture method on an image processing apparatus and an image processing method able to reduce a reflection in a captured image are reduced, while PARK there remains a need for a method and system to record a video memory that includes both the photographer and the scene participants at the same time. Please see NAKAGAWA (US 20220253993 A1), Paragraph [0005], and PARK et al. (US 20130235223 A1), Paragraph [0003].
Regarding claim 19, NAKAGAWA explicitly teaches a method comprising (Fig. 6A-6B, illustrate a method. Paragraph [0071]-NAKAGAWA discloses the operations of the CPU 101 relating to image capture in the reflection reduction mode will be described using the flowchart illustrated in FIGS. 6A and 6B.):
obtaining, by a device (Fig. 1, #100 called a smart phone with components #101 CPU, #102 memory, #107 outward-facing camera, #109, inward facing camera, #105 display. Paragraph [0022]-NAKAGAWA discloses the CPU 101 controls the operations of the functional blocks by loading programs (the OS, applications, and the like) stored in a non-volatile memory 103 into a memory 102 and executing them to realize the various functions of the smart phone 100. Further in paragraph [0025]-NAKAGAWA discloses an image processing circuit 104 is controlled by the CPU 101 to apply various types of image processing to the image data obtained via image capture by an outward-facing camera 107, a standard inward-facing camera 109, and a wide-angle inward-facing camera 110 or the image data stored in the non-volatile memory 103. Further in paragraph [0034]-NAKAGAWA discloses a display 105 is a touch liquid crystal display, for example. The display 105 is controlled by the CPU 101 to display images (icons, windows, software keys, and the like) of the GUI of the OS or applications, images captured by a camera, and the like.), first image data representing a first image captured by a first camera facing a first direction (Fig. 2B, illustrates a first camera called outward-facing camera #107 (wherein outward-facing is a first direction). Paragraph [0025]-NAKAGAWA discloses an image processing circuit 104 is controlled by the CPU 101 to apply various types of image processing to the image data obtained via image capture by an outward-facing camera 107.);
obtaining, by the device (Fig. 1, #100 called a smart phone with components #101 CPU, #102 memory, #107 outward-facing camera, #109, inward facing camera, #105 display. Paragraph [0022]-NAKAGAWA discloses the CPU 101 controls the operations of the functional blocks by loading programs (the OS, applications, and the like) stored in a non-volatile memory 103 into a memory 102 and executing them to realize the various functions of the smart phone 100. Further in paragraph [0025]-NAKAGAWA discloses an image processing circuit 104 is controlled by the CPU 101 to apply various types of image processing to the image data obtained via image capture by an outward-facing camera 107, a standard inward-facing camera 109, and a wide-angle inward-facing camera 110 or the image data stored in the non-volatile memory 103. Further in paragraph [0034]-NAKAGAWA discloses a display 105 is a touch liquid crystal display, for example. The display 105 is controlled by the CPU 101 to display images (icons, windows, software keys, and the like) of the GUI of the OS or applications, images captured by a camera, and the like.), second image data representing a second image captured by a second camera facing a second direction that is opposite to the first direction (Fig. 2A-2B. Paragraph [0025]-NAKAGAWA discloses an image processing circuit 104 is controlled by the CPU 101 to apply various types of image processing to the image data obtained via image capture by an outward-facing camera 107, a standard inward-facing camera 109, and a wide-angle inward-facing camera 110 (wherein the inward-facing camera is a direction opposite of the outward facing camera).);
identifying, by the device (Fig. 1, #100 called a smart phone with components #101 CPU, #102 memory, #107 outward-facing camera, #109, inward facing camera, #105 display. Paragraph [0022]-NAKAGAWA discloses the CPU 101 controls the operations of the functional blocks by loading programs (the OS, applications, and the like) stored in a non-volatile memory 103 into a memory 102 and executing them to realize the various functions of the smart phone 100. Further in paragraph [0025]-NAKAGAWA discloses an image processing circuit 104 is controlled by the CPU 101 to apply various types of image processing to the image data obtained via image capture by an outward-facing camera 107, a standard inward-facing camera 109, and a wide-angle inward-facing camera 110 or the image data stored in the non-volatile memory 103. Further in paragraph [0034]-NAKAGAWA discloses a display 105 is a touch liquid crystal display, for example. The display 105 is controlled by the CPU 101 to display images (icons, windows, software keys, and the like) of the GUI of the OS or applications, images captured by a camera, and the like.) and based on the second image data, a region in the first image that includes one or more reflected objects (Fig. 4A. Paragraph [0043]-NAKAGAWA discloses using the image of the person capturing the image captured by the wide-angle inward-facing camera 110, reflections in the image of the subject captured by the outward-facing camera 107 are reduced. Further in paragraph [0061]-NAKAGAWA discloses the CPU 101 instructs the image processing circuit 104 to detect if there is a reflection of the user based on the image obtained by the inward-facing camera 110 and the image obtained by the outward-facing camera 107. The image processing circuit 104 uses known technology such as pattern matching to detect if there is a reflection of the user and notifies the CPU 101 of the result.);
NAKAGAWA fails to explicitly teach generating, by the device, fill-in image data based on the first image data, the fill-in image data representing a fill-in image; and generating, by the device, output image data, based on the first image data and the fill-in image data, that corresponds to the first image in which the region is replaced with the fill-in image.
However, CHANG explicitly teaches generating, by the device (Fig. 1, #10 called electronic apparatus. Paragraph [0019-0021]-CHANG discloses the electronic apparatus 10 includes an image retrieving unit 12, a storage unit 14, and a processing unit 16. The electronic apparatus 10 is, for example, a portable electronic device, such as a camera, a smartphone, a personal digital assistant (PDA), a tablet or the like, and the application is not limited thereto. The image retrieving unit 12 is, for example, a photosensitive element for capturing images such as a Charge Coupled Device (CCD), a Complementary Metal-Oxide Semiconductor (CMOS), or other elements, and the application is not limited thereto. The storage unit 14 is, for example, one or a combination of a stationary or portable random access memory (RAM), read-only memory (ROM), flash memory, hard disk, or any other similar device, and which records a plurality of modules that can be executed by the processing unit 16. Further paragraph [0022]-CHANG discloses the processing unit 16 is, for example, a Central Processing Unit (CPU), a programmable microprocessor, a programmable controller, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Programmable Logic Device (PLD), or other devices, and is coupled to the image retrieving unit 12 and the storage unit 14, for processing the reflection in an image obtained by the image retrieving unit 12. Further in paragraph [0024]-CHANG discloses the images being obtained by the image retrieving unit 12 may be preview images and are displayed on a screen (not shown) of the electronic apparatus 10, so as to assist the user in finding a view for photographing.), fill-in image data based on the first image data (Fig. 1. Paragraph [0031]-CHANG discloses it is noted that, in another embodiment, an image inpainting algorithm may be further adopted to remove the reflection recognized in the foregoing embodiment. In detail, after the reflection determining module 148 determines which object is the reflection in the step S210 in aforesaid embodiment, a reflection processing module (not shown) stored in the storage unit 14 may be further executed by the processing unit 16 to remove the reflection from the image and inpaints a removed region of the image with other images by means of image inpainting (wherein inpainting is generating fill-in image data).), the fill-in image data representing a fill-in image (Fig. 1. Paragraph [0032]-CHANG discloses through the technique of image inpainting, the removed region of the image can be recovered by other images obtained by the image retrieving unit 12 (wherein the other images are fill-in images).); and
generating, by the device (Fig. 1, #10 called electronic apparatus. Paragraph [0019-0021]-CHANG discloses the electronic apparatus 10 includes an image retrieving unit 12, a storage unit 14, and a processing unit 16. The electronic apparatus 10 is, for example, a portable electronic device, such as a camera, a smartphone, a personal digital assistant (PDA), a tablet or the like, and the application is not limited thereto. The image retrieving unit 12 is, for example, a photosensitive element for capturing images such as a Charge Coupled Device (CCD), a Complementary Metal-Oxide Semiconductor (CMOS), or other elements, and the application is not limited thereto. The storage unit 14 is, for example, one or a combination of a stationary or portable random access memory (RAM), read-only memory (ROM), flash memory, hard disk, or any other similar device, and which records a plurality of modules that can be executed by the processing unit 16. Further paragraph [0022]-CHANG discloses the processing unit 16 is, for example, a Central Processing Unit (CPU), a programmable microprocessor, a programmable controller, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Programmable Logic Device (PLD), or other devices, and is coupled to the image retrieving unit 12 and the storage unit 14, for processing the reflection in an image obtained by the image retrieving unit 12. Further in paragraph [0024]-CHANG discloses the images being obtained by the image retrieving unit 12 may be preview images and are displayed on a screen (not shown) of the electronic apparatus 10, so as to assist the user in finding a view for photographing.), output image data, based on the first image data and the fill-in image data, that corresponds to the first image in which the region is replaced with the fill-in image (Fig. 1. Paragraph [0036]-CHANG discloses the recognized reflection can be further removed and the removed region can be inpainted with other images or neighboring regions through image inpainting. As a result, an image without reflection can be obtained and a better image quality is achieved. (wherein the input image is the first image, the region that is inpainted is the fill-in image data, and the output image data is the image without reflection).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of NAKAGAWA of a method comprising: obtaining, by a device, first image data representing a first image captured by a first camera facing a first direction; obtaining, by the device, second image data representing a second image captured by a second camera facing a second direction that is opposite to the first direction; identifying, by the device and based on the second image data, a region in the first image that includes one or more reflected objects; with the teachings of CHANG of generating, by the device, fill-in image data based on the first image data, the fill-in image data representing a fill-in image; and generating, by the device, output image data, based on the first image data and the fill-in image data, that corresponds to the first image in which the region is replaced with the fill-in image.
Wherein having NAKAGAWA’s method/device of image reflection removal having generating, by the device, fill-in image data based on the first image data, the fill-in image data representing a fill-in image; and generating, by the device, output image data, based on the first image data and the fill-in image data, that corresponds to the first image in which the region is replaced with the fill-in image.
The motivation behind the modification would have been to obtain method/device capable of efficiently and accurately removing reflections from images. Since both NAKAGAWA and CHANG relate to removing reflections from images, wherein NAKAGAWA limitations relating to capturing scenes and image capture method on an image processing apparatus and an image processing method able to reduce a reflection in a captured image are reduced, while CHANG the nearby object is recognized as a reflection and can be removed from the image so as to improve the image quality. Please see NAKAGAWA (US 20220253993 A1), Paragraph [0005], and CHANG et al. (US 20150363920 A1), Paragraph [0010].
Although NAKAGAWA in view of CHANG explicitly teach the device. NAKAGAWA in view of CHANG are silent on having the device with a modem and speaker based on the 112(f) claim interpretation.
However, PARK explicitly teaches a device with a modem and speaker (Fig. 1, #10 called the digital camera with #28 firmware memory, #20 processor, #14 first image sensor, #15 second image sensor, #32 image display, #26 speaker, #50 wireless modem, and #34 user controls. Paragraph [0041]-PARK discloses the forward-facing lens 4 focuses light from a scene (not shown) onto a first image sensor 14, for example, a single-chip color CCD or CMOS image sensor. The forward-facing lens 4 is one type optical system for forming an image of the scene on the first image sensor 14. The rear-facing lens 5 focuses light from a scene (not shown) onto a second image sensor 15. The first image sensor 14 and the second image sensor can be, for example, single-chip color CCDs or CMOS image sensors (wherein an image sensor is a camera). Further in paragraph [0041]-PARK discloses the processor 20 includes a program memory (not shown), and the software programs stored in the firmware memory 28 are copied into the program memory before being executed by the processor 20. Further in paragraph [0054]-PARK discloses the processor 20 produces menus and low resolution color images that are temporarily stored in display memory 36 and are displayed on image display 32. Further in paragraph [0056]-PARK discloses some of the user controls 34 are provided by using a touch sensitive surface, such a touch screen overlay on the image display 32. Further in paragraph [0058]-PARK discloses an audio codec 22 connected to the processor 20 receives an input audio signal from a forward-facing microphone 24 and provides an output audio signal to a speaker 26. Further in paragraph [0063]-PARK discloses the wireless modem 50 communicates over a radio frequency (e.g. wireless) link with a mobile phone network (not shown), such as a 3GSM network, which connects with the Internet 70 in order to upload digital image files from the digital camera 10.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of NAKAGAWA in view of CHANG of a method comprising: obtaining, by a device, first image data representing a first image captured by a first camera facing a first direction; obtaining, by the device, second image data representing a second image captured by a second camera facing a second direction that is opposite to the first direction; identifying, by the device and based on the second image data, a region in the first image that includes one or more reflected objects; with the teachings of PARK of a device with a memory, a processor, a first camera, a second camera, a display, a speaker, a modem, and a touchscreen.
Wherein having NAKAGAWA’s method/device of image reflection removal having a device with a memory, a processor, a first camera, a second camera, a display, a speaker, a modem, and a touchscreen.
The motivation behind the modification would have been to obtain method/device capable of efficiently and accurately removing reflections from images that has widescale fast internet connection. Since both NAKAGAWA and PARK relate to using opposing cameras to analyze specific regions of an image, wherein NAKAGAWA limitations relating to capturing scenes and image capture method on an image processing apparatus and an image processing method able to reduce a reflection in a captured image are reduced, while PARK there remains a need for a method and system to record a video memory that includes both the photographer and the scene participants at the same time. Please see NAKAGAWA (US 20220253993 A1), Paragraph [0005], and PARK et al. (US 20130235223 A1), Paragraph [0003].
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over NAKAGAWA (US 20220253993 A1), hereinafter referenced as NAKAGAWA, in view of CHANG et al. (US 20150363920 A1), hereinafter referenced as CHANG, and further in view of TSAI et al. (US 11276177 B1), hereinafter referenced as TSAI.
Regarding claim 17, NAKAGAWA in view of CHANG explicitly teach the device of claim 1,
NAKAGAWA further explicitly teaches wherein the one or more processors are integrated in at least one of a mobile phone, a tablet computer device, or a camera device (Fig. 1. Paragraph [0017]-NAKAGAWA discloses the present invention is describing using the example of a smart phone provided with a camera (inward-facing camera) facing inward (the side of the person capturing the image) and a camera (outward-facing camera) facing outward (the side of the subject). However, the present invention is able to be implemented as other electronic devices that are provided with an inward-facing camera and an outward-facing camera, such as a tablet terminal and a laptop computer (wherein a smartphone with an inward and outward facing camera is a camera device).), and
wherein the mobile phone, the tablet computer device, or the camera device (Fig. 1. Paragraph [0017]-NAKAGAWA discloses the present invention is describing using the example of a smart phone provided with a camera (inward-facing camera) facing inward (the side of the person capturing the image) and a camera (outward-facing camera) facing outward (the side of the subject). However, the present invention is able to be implemented as other electronic devices that are provided with an inward-facing camera and an outward-facing camera, such as a tablet terminal and a laptop computer (wherein a smartphone with an inward and outward facing camera is a camera device).) is configured to initiate display of an output image based on the output image data (Fig. 1, #105 called display. Paragraph [0035]-NAKAGAWA discloses the CPU 101 generates display data and stores the display data in a video memory area of the memory 102. The display 105 generates a video signal on the basis of the display data and displays the video signal on a display panel).
NAKAGAWA in view of CHANG fail to explicitly teach a wearable electronic device, the wearable electronic device.
However, TSAI explicitly teaches a wearable electronic device (Fig. 1. Col. 8, Line [25-26]-TSAI discloses the image processing system 100 can be part of an electronic device (or devices) such as a camera system (e.g., a digital camera, an IP camera, a video camera, a security camera, etc.), a telephone system (e.g., a smartphone, a cellular telephone, a conferencing system, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a display device, a digital media player, a gaming console, a video streaming device, a drone, a computer in a car, an IoT (Internet-of-Things) device, a smart wearable device, an extended reality (XR) device (e.g., a head-mounted display, smart glasses, etc.), or any other suitable electronic device(s).),
the wearable electronic device (Fig. 1. Col. 8, Line [25-26]-TSAI discloses the image processing system 100 can be part of an electronic device (or devices) such as a camera system (e.g., a digital camera, an IP camera, a video camera, a security camera, etc.), a telephone system (e.g., a smartphone, a cellular telephone, a conferencing system, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a display device, a digital media player, a gaming console, a video streaming device, a drone, a computer in a car, an IoT (Internet-of-Things) device, a smart wearable device, an extended reality (XR) device (e.g., a head-mounted display, smart glasses, etc.), or any other suitable electronic device(s).),
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of NAKAGAWA in view of CHANG and further in view of KIM of a device comprising: a memory configured to store first image data representing a first image captured by a first camera facing a first direction; and one or more processors, coupled to the memory, wherein the one or more processors are configured to: obtain second image data representing a second image captured by a second camera facing a second direction that is opposite to the first direction; identify, based on the second image data, a region in the first image that includes one or more reflected objects; with the teachings of TSAI of a wearable electronic device, the wearable electronic device.
Wherein having NAKAGAWA’s method/device of image reflection removal having a wearable electronic device, the wearable electronic device.
The motivation behind the modification would have been to obtain method/device capable of efficiently and accurately removing reflections from images. Since both NAKAGAWA and TSAI relate to analyzing regions in images captured by cameras with different FOVs, wherein NAKAGAWA limitations relating to capturing scenes and image capture method on an image processing apparatus and an image processing method able to reduce a reflection in a captured image are reduced, while TSAI the technologies described herein can leverage images with different field-of-views (FOVs) and/or captured using different types of lenses to improve and refine segmentation results. Please see NAKAGAWA (US 20220253993 A1), Paragraph [0005], and TSAI et al. (US 11276177 B1), Col. 6, Lines [51-62].
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over NAKAGAWA (US 20220253993 A1), hereinafter referenced as NAKAGAWA, in view of CHANG et al. (US 20150363920 A1), hereinafter referenced as CHANG, and further in view of KIM et al. (US 20200167943 A1), hereinafter referenced as KIM.
Regarding claim 18, NAKAGAWA in view of CHANG explicitly teach the device of claim 1,
NAKAGAWA in view of CHANG fail to explicitly teach wherein the one or more processors are integrated in a vehicle that is configured to initiate display of an output image based on the output image data.
However, KIM explicitly teaches wherein the one or more processors are integrated in a vehicle (Fig. 3. Paragraph [0095]-KIM discloses the PPU 300 may be included in a desktop computer, a laptop computer, a tablet computer, servers, supercomputers, a smart-phone (e.g., a wireless, hand-held device), personal digital assistant (PDA), a digital camera, a vehicle, a head mounted display, a hand-held electronic device, and the like.) that is configured to initiate display of an output image based on the output image data (Fig. 3. Paragraph [0055]-KIM discloses the PPU 300 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data in order to generate two-dimensional (2D) image data for display on a display device such as a liquid crystal display (LCD) device.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of NAKAGAWA in view of CHANG of a device comprising: a memory configured to store first image data representing a first image captured by a first camera facing a first direction; and one or more processors, coupled to the memory, wherein the one or more processors are configured to: obtain second image data representing a second image captured by a second camera facing a second direction that is opposite to the first direction; identify, based on the second image data, a region in the first image that includes one or more reflected objects; with the teachings of KIM of wherein the one or more processors are integrated in a vehicle that is configured to initiate display of an output image based on the output image data.
Wherein having NAKAGAWA’s method/device of image reflection removal having generate a segmentation mask based on the second image; and.
The motivation behind the modification would have been to obtain method/device capable of efficiently and accurately removing reflections from images. Since both NAKAGAWA and KIM relate to analyzing specific regions in an image to identify subjects, wherein NAKAGAWA limitations relating to capturing scenes and image capture method on an image processing apparatus and an image processing method able to reduce a reflection in a captured image are reduced, while KIM states approaches suffer from some limitations such as: (1) missing small surfaces; (2) requiring a maximum number of planes in a single image, a priori; and (3) poor generalization across domains. Therefore, there is a need for addressing these issues and/or other issues associated with the prior art. Please see NAKAGAWA (US 20220253993 A1), Paragraph [0005], and KIM et al. (US 20200167943 A1), Paragraph [0003].
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant’s disclosure.
KEE et al. (US 20250117994 A1) - In implementation of techniques for removing image overlays, a computing device implements a reflection removal system to receive an input RAW digital image, the input RAW digital image including both a base image and an overlay image. Using a machine learning model, the reflection removal system segments the base image from the overlay image. The reflection removal system generates an output RAW digital image that includes the base image and displays the output RAW digital image in a user interface…Abstract, Fig. 1 and 4.
ONO et al. (US 20230410464 A1) - A black-and-white polarized image acquirer 21 of an image acquirer 20 acquires a black-and-white polarized image, and a color image acquirer 22 acquires a color image. A reflection-removed image generator 40 calculates characteristic information related to black-and-white reflection removal results generated based on a black-and-white polarized image. For example, the reflection-removed image generator calculates information indicating a relationship between the black-and-white polarized image and an average luminance calculated from an image for each polarization direction in the black-and-white polarized image, or information used for obtaining the black-and-white reflection removal results from the black-and-white polarized image. The reflection-removed image generator 40 generates a reflection-removed color image in which specular reflection is removed from the color image using the calculated characteristic information or the optimized characteristic information. Thus, a high-sensitivity reflection-removed color image is obtained…Abstract, Fig. 8.
KHIZBULLIN et al. (US 20240005464 A1) – The technology of this application relates to a method for removing reflections from an image. The method detects one or more reflection areas in the image, wherein each reflection area includes a reflection. Further, the method extracts the one or more reflection areas from the image, and removes the reflection from each of the extracted reflection areas…Abstract, Fig. 4-5.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ETHAN N WOLFSON whose telephone number is (571)272-1898. The examiner can normally be reached Monday - Friday 8:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ETHAN N WOLFSON/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights