DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Foreign priority is claimed to INDIA 202241027494 05/12/2022 and INDIA 202241027494 10/12/2022. They have been accompanied by certified copies of received as required by 37 CFR 1.55.
Information Disclosure Statement
The IDS(s) has/have been considered and placed in the application file.
Response to Arguments
This final office action is in response to the amendment filed September 29, 2026. Claims 1-6 and 8-18 are pending in this application and have been considered below. Claims 7, 19, and 20 are canceled by the applicant.
Applicant’s arguments with respect to claims 1 and 13 have been considered but are moot in view of new ground(s) of rejection because of the amendments. Lim remains relied upon for determining the shadow, which it does by color difference before any ToF measurement (Lim: ¶¶ 148-149), for the two-camera capture, and for the removal; Gumpert is relied on for the optical zoom level that corresponds to a size of the shadow; Wang is relied upon only for the two AI models and the shadow mask, not for camera settings.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 2, 4-6, 8, 13, 14, and 16-18 are rejected under 35 U.S.C. 103 as being unpatentable over Lim et al., WO 2022/010193 A1 (hereinafter “Lim”) in view of Gumpert et al., US 2018/0376074 A1 (hereinafter “Gumpert”).
Claim 1.
Lim in view of Gumpert teaches a method for removing a shadow from an image, comprising:
obtaining, by a multi-camera device, a first image of an input scene from a first camera of the multi-camera device (Lim: ¶¶ 120-122. The first camera module 342 is a wide-angle camera that acquires the first image of the subject scene and displays it on a first preview screen.);
determining, by the multi-camera device, a shadow in the first image, and at least one property and a region of interest (ROI) of the shadow in the first image (Lim: ¶ 123-134, 146-148. The device determines whether the first image contains a shadow region, the shadow area of the first image and a color difference value, at the subject boundary; the first image area 910 containing the shadow is the ROI, and the shadow are and the color difference values are properties.);
applying, by the multi-camera device, an optical zoom level and an exposure time to a second camera for obtaining a second image of the input scene (Lim: ¶ 126-127, 158. Upon determining that image improvement is required, the device obtains a second image using the second camera module 344, a telephoto camera, which captures subject detail not obtained by the wide-angle first camera.); and
removing, by the multi-camera device, at least one portion of the shadow from the first image, based on the first image and the second image (Lim: ¶ 134-135. The first and second images are matched by object coordinates and synthesized into a composite image from which the shadow region is removed.).
Lim does not state that the optical zoom level and the exposure time applied to the second camera are selected based on the determined shadow property and ROI, or that the optical zoom level corresponds to the size of the shadow. Lim determines a value of the shadow area and a color difference value at the subject boundary in the first image area 910, which contains the shadow (Lim: ¶¶ 124, 145-148.), provides a second camera with optical zoom (Lim: ¶ 51.) and exposure time (Lim: ¶ 54.), and obtains the second image in response to detecting the shadow to produce the improved image Lim seeks (Lim: ¶¶ 126-127, 158.).
Gumpert is in the same field of endeavor of image capturing devices, has a wide-view first camera and a second zoom camera (Gumpert ¶ 3), and is reasonably pertinent to selecting a second camera’s zoom for a region detected in the first camera’s image (SPEC ¶ 94) (MPEP 2141.01(a)); it teaches setting the optical zoom level of a second camera according to the size of a region detected in the wider view of a first camera (Gumpert ¶ 32: “a zoom motor enabling automatic ( or user controlled) optical zoom”; ¶ 37: “enabling a close-up view of one or more detected objects, and the size data to enable use of a suitable zoom setting”; ¶ 44: “the control unit may generate new data concerning the size of the super-object, resulting in a shift of the zoom settings of the PTZ-camera.”).
It would have been obvious before the effective filing date of the claimed invention to select the optical zoom level and exposure time of the second camera as a function of the shadow property or ROI that Lim already determines, so that the second image captures additional context sufficient for the image improvement that is Lim's stated objective. This is a combination of Lim's known shadow determination with Lim's known capture settings according to their established functions to yield the predictable result that the second image is matched to the detected shadow (MPEP 2143(A)).
It would have been obvious before the effective filing date of the claimed invention to give Lim’s second camera Gumpert’s optical zoom, to estimate the size of the shadow region that Lim detects (Lim: ¶¶ 124, 146), as Gumpert estimates the size of a detected region (Gumpert ¶ 36: “position, estimated size”), and to set the optical zoom level according to that size, to help the second image include the whole shadowed area of the scene while increasing the detail captured there for the removal (Lim: ¶ 135), which follows Gumpert's zoom setting (Gumpert ¶ 12: “as much as possible while still including the whole superobject in its field of view”) with predictable results (MPEP 2143(1)(C). KSR lnt'I Co. v. Teleflex Inc., 550 U.S. 398, 417 (2007).
The Examiner took Official Notice in the Office action mailed June 30, 2026 that capturing a darker region of a scene with a longer exposure time, to admit more light and recover color and texture detail in that region, is well known and conventional in photography. Applicant's statement that they “respectfully traverses the rejections” does not specifically point out any error in the noticed fact, so the traverse is inadequate and the statement is taken to be admitted prior art. MPEP 2144.03(C). It would have been obvious before the effective filing date of the claimed invention to set the exposure time that Lim's image signal processor controls (Lim: ¶¶ 54-55.) for the second camera according to how dark the shadow region is and to that region (the ROI), to help recover color and texture detail in the shadow region for the removal, with predictable results (MPEP 2143(I)(D)).
Claims 2 and 14.
Lim in view of Gumpert teaches wherein the at least one property of the at least one shadow comprises at least one of a shadow intensity, a shadow complexity, shadow area, or a shadow type (Lim:¶ 124. The device determines the shadow area of the first image.). Claim 14 recites the multi-camera device counterpart of claim 2 and is rejected for the same reasons, mutatis mutandis.
Claims 4 and 16.
Lim in view of Gumpert teaches wherein the second camera comprises at least one of a wide angle camera, a telephoto camera, an optical zoom camera, a standard camera, or an ultra-wide camera (Lim: ¶ 127, 51. The second camera module 344 is a telephoto camera; the lens assembly properties include optical zoom and a telephoto lens.). Claim 16 recites the multi-camera device counterpart of claim 4 and is rejected for the same reasons, mutatis mutandis.
Claims 5 and 17.
Lim in view of Gumpert teaches wherein the at least one additional context comprises at least one of lighting conditions, object colors, finer details, or textures of the input scene (Lim: ¶ 127. The telephoto second camera captures subject detail not obtained by the wide-angle first
camera i.e. greater magnification yields finer details). Claim 17 recites the multi-camera device counterpart of claim 5 and is rejected for the same reasons, mutatis mutandis.
Claims 6 and 18.
Lim in view of Gumpert teaches a system user interface and a display manager that output graphical user interface elements on the display (Lim: ¶ 96, 104. The system UI 515 configures GUI screens and the display manager 526 manages output of GUI elements.), but does not state providing an icon for the removing the shadow based on a shadow removing event.
The Examiner took Official Notice in the Office action mailed June 30, 2026 that providing an icon as a graphical user interface control to invoke a function and indicate its status is well known and conventional (MPEP 2144.03). Applicant’s statement that it “respectfully traverses the rejection” does not specifically point out any error in the noticed fact, so the traverse is inadequate and the statement is taken to be admitted prior art. MPEP 2144.03(C). It would have been obvious before the effective filing date of the claimed invention to provide such an icon among Lim's graphical user interface elements to invoke and indicate the shadow removal function. The motivation would have been to give the user a direct control to trigger shadow removal and to convey that the function is active, a predictable use of a known graphical user interface element.
Claim 8.
Lim in view of Gumpert teaches removing the at least one portion of the shadow from the first image, based on the at least one property of the at least one shadow and the ROI obtained from the first image, or the at least one additional context obtained from the second image (Lim: ¶ 124, 134-135. The shadow region determined from the first image and the synthesized telephoto second image content are used to produce the shadow-free composite.).
Claim 13.
Lim in view of Gumpert teaches a multi-camera device comprising: a first camera and a second camera (Lim: ¶ 120. Camera module 340 includes a first camera module 342 (wide-angle) and a second camera module 344 (telephoto).); memory storing instructions (Lim: ¶ 118, 188. Memory
330 stores instructions that are executed by the processor.); and at least one processor operatively connected to the memory, the first camera, and the second camera, wherein the instructions, when executed by the at least one processor is configured to cause the multi-camera device to (Lim: ¶ 64, 119. Processor 310 is connected to and controls memory 330 and camera module 340.): obtain a first image of an input scene from the first camera (Lim: ¶ 120-122); determine a shadow in the first image, and at least one property and a region of interest (ROI) of the shadow in the first image (Lim: ¶ 123-124, 146-148); apply an optical zoom level and an exposure time to the second camera for obtaining a second image of the input scene (Lim: ¶¶ 126-127, 158); and remove at least one portion of the shadow from the first image, based on the first image and the second image (Lim: ¶ 134-135). The optical zoom level and exposure time limitations are rejected for the reasons given for claim 1, mutatis mutandis.
Claims 3, 9-12 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Lim in view of Gumpert as applied to claims 1, 6, and 13 above, and further in view of Wang et al., "Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal," (hereinafter “Wang”).
Claims 3 and 15.
Lim in view of Gumpert teaches determining the region of interest of the shadow (Lim:¶ 124, 147-148. The device determines the shadow area and a color difference value at the subject boundary.), but does not state that the ROI is a shadow mask which indicates an area of the shadow.
Representing a determined shadow region as a shadow mask is a conventional manner of delineating the shadow area for processing, and is further taught by Wang, whose first generator produces a shadow detection mask indicating the shadow region (Wang:
Abstract.).
It would have been obvious before the effective filing date of the claimed invention
to represent Lim's shadow region as a shadow mask indicating the area of the shadow, in order
to provide a standard, computationally convenient delineation of the shadow region for the
subsequent removal operation, a predictable result.
Claim 9.
Lim in view of Gumpert teaches the method of claim 6 and further teaches applying the optical zoom level of the second camera and selecting the optical zoom level of the second camera for obtaining the second image to identify the additional context (Lim: ¶ 51, 126-127, 158. The lens assembly properties include optical zoom; the telephoto second camera captures detail of the subject not obtained by the wide-angle first camera.), and analyzing the property or ROI of the first image and the additional context of the second image and removing the shadow based on the first image and the second image (Lim:¶ 134-135.). Selection of the optical zoom level based on the shadow property and ROI is taught for the reasons given for claim 1.
Lim determines and removes the shadow through color-difference detection, time-of-flight distance measurement, object-coordinate matching, and image synthesis. Lim does not state determining the at least one property of the shadow and the ROI using a first AI model, or removing the at least one shadow using a second AI model. Wang teaches a shadow framework of two stacked conditional generative adversarial networks, in which a shadow image is fed into a first generator that produces a shadow detection mask, and the shadow image concatenated with the predicted mask is fed into a second generator that recovers a shadow-free image (Wang: Abstract; see also Fig. 2. "a shadow image is fed into the first generator which produces a shadow detection mask. That shadow image, concatenated with its predicted mask, goes through the second generator in order to recover its shadow-free image consequently."). Wang's first generator is a first AI model that determines the ROI of the shadow as the shadow detection mask, and Wang's second generator is a second AI model that removes the shadow. Wang’s shadow detection mask also indicates the area of the shadow, which claim 2 lists as a property of the shadow, so the first AI model determines both the at least one property and the ROI.
It would have been obvious before the effective filing date of the claimed invention to implement the shadow detection and the shadow removal of Lim using the first AI model and the second AI model of Wang, as a substitution of one known shadow detection and removal technique for another known technique to obtain the predictable result of detecting and removing the shadow (MPEP 2143 (B)). Lim's multi-camera capture of the first image and the second image is retained and supplies the input to the models, so that the first AI model determines the shadow property and ROI from the first image and the second AI model removes the shadow based on the first image and the additional context of the second image; Lim's principle of operation of capturing complementary images to improve the shadow region is therefore unchanged. A person of ordinary skill would have had a reasonable expectation of success because Wang demonstrates that the two stacked models successfully detect and remove shadows. The motivation would have been to obtain the improved detection and removal accuracy that Wang teaches results from jointly learning the two tasks (Wang: Abstract.).
Claim 10.
Lim in view of Gumpert and Wang teaches optical zoom as a lens property and switching to the telephoto second camera (Lim: ¶ 51, 158). It would have been obvious to select the optical zoom level from a zoom-in or a zoom-out, as Gumpert’s camera does (Gumpert ¶ 29: “zooming in to achieve greater magnification … zooming out to dynamically adjust to changes in the
scene.”) to frame the additional capture of the shadow region.
Claim 11.
Claim 11 recites applying the exposure time of the second camera. Lim in view of Gumpert teaches exposure time control of the camera and switching to the second camera to obtain the additional context (Lim: ¶¶ 54, 158. The image signal processor 260 controls exposure time.). The first AI model and the second AI model are taught by Wang (Wang: Abstract and Fig. 2.) as set forth for claim 9, and the same rationale and motivation to combine apply, mutatis mutandis. Varying the exposure time on the shadow property and ROI is taught for the reasons given for claim 1.
Claim 12.
Lim in view of Gumpert and Wang teaches the method of claim 11. Lim teaches controlling the exposure time (Lim: ¶ 54.). It would have been obvious that the varying of the exposure time comprises adjusting the time range from a lower to higher range or a higher to lower range, these being the two available directions of any exposure-time adjustment, in order to admit more or less light to recover detail in the shadow region, a predictable result.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ross Varndell whose telephone number is (571)270-1922. The examiner can normally be reached M-F, 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’Neal Mistry can be reached at (313)446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Ross Varndell/Primary Examiner, Art Unit 2674