DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Foreign priority is claims to INDIA 202241027494 05/12/2022 and INDIA 202241027494 10/12/2022. However, no certified copies of these papers have been received as required by 37 CFR 1.55.
Information Disclosure Statement
The IDS(s) has/have been considered and placed in the application file.
Information Disclosure Statement
The IDSs has been considered and placed of record in file. One of the NPL references on the IDS filed November 12, 2024, was illegible and not considered. If the reference would like to be considered please submit another IDS form with a legible copy of the NPL.
Claim Objections
Claim 18 is objected to because it recites that the at least one processor is "further configured to execute the instructions to: providing an icon." The gerund "providing" following "to" renders the limitation grammatically incorrect. It is suggested that "providing" be amended to "provide."
Claim Rejections- 35 U.S.C. § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION-The specification shall conclude with one or more claims particularly
pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 9-12 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to
particularly point out and distinctly claim the subject matter which the inventor regards as the
invention.
Claims 9 and 11 each recite "the at least one of the at the least one property of the at least one shadow. " The phrase "the at the least one property" is grammatically incoherent, and the metes and bounds of the limitation cannot be determined. Correction is required.
Claim 12 depends from claim 11 and is rejected for the same reason, as it incorporates the indefinite limitation of claim 11.
Claim 10 recites "selecting the optical zoom level from at least one of a zoom-in or a zoom-out." There is insufficient antecedent basis for "the optical zoom level" in the claim. Neither claim 10, parent claim 8, nor claim 1 previously introduces an optical zoom level.
Claim 11 recites "determining, by the multi-camera device, the at least one of the at least one property of the at least one shadow or the ROI of the at least one shadow in the first image captured by the first camera, using the first AI model." There is insufficient antecedent basis for "the first AI model" in the claim. Claim 11 depends from claim 6, and neither claim 11, parent claim 6, nor claim 1 previously introduces a first AI model. The term "a first Artificial
Intelligence (Al) model" is first introduced in claim 9, from which claim 11 does not depend.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 2, 4-8, 10, and 13, 14, and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over
Lim et al., WO 2022/010193 A1 (hereinafter “Lim”), published January 13, 2022. Lim qualifies as prior art under 35 U.S.C. 102(a)(1) because it is a printed publication that published on January 13, 2022, before the May 12, 2022 effective filing date. Because Lim is applied as prior art under 35 U.S.C. 102(a)(1), the exception of 35 U.S.C. 102(b)(2)(C) does not apply: by its terms that exception removes a disclosure only as prior art under subsection (a)(2), and it therefore has no effect on a printed publication applied under 35 U.S.C. 102(a)(1). The common assignment of Lim and the present application to a single entity accordingly does not remove Lim as prior art. Citations are to the English-language machine translation of Lim made of record herewith; paragraph numbers correspond to the published WO document.
Claim 1.
Lim teaches a method for removing at least one shadow from an image, comprising:
receiving, by a multi-camera device, a first image of an input scene from a first camera of the multi-camera device (Lim: ¶¶ 120-122. The first camera module 342 is a wide-angle camera that acquires the first image of the subject scene and displays it on a first preview screen.);
identifying, by the multi-camera device, at least one shadow in the first image (Lim: ¶¶ 123-
124, 146-148. The device determines whether the first image contains a shadow region
generated by the subject or photographer, including by a color difference value at the
subject boundary.);
determining, by the multi-camera device, at least one of at least one property of the at least one shadow or a region of interest (ROI) of the at least one shadow in the first image (Lim: ¶ 124, 147-148. The device determines the shadow area of the first image and a color difference value of the shadow region.);
applying, by the multi-camera device, at least one configuration to a second camera for obtaining a second image of the input scene (Lim: ¶ 126-127, 158. Upon determining that image improvement is required, the device obtains a second image using the second camera module 344, a telephoto camera, which captures subject detail not obtained by the wide-angle first camera.); and
removing, by the multi-camera device, the at least one shadow from the first image, based on the first image and the second image (Lim: ¶ 134-135. The first and second images are matched by object coordinates and synthesized into a composite image from which the shadow region is removed.).
Lim does not state that the at least one configuration applied to the second camera is selected based on the determined shadow property or ROI. Lim quantifies the shadow, determining the shadow area and a color-difference value of the shadow region (Lim: ¶ 124, 147-148.), provides a second camera with adjustable optical zoom (Lim: ¶ 51.) and exposure time (Lim: ¶ 54.), and obtains the second image in response to detecting the shadow to produce the improved image Lim seeks (Lim: ¶¶ 126-127, 158.).
It would have been obvious before the effective filing date of the claimed invention to select the adjustable configuration of the second camera as a function of the shadow property or ROI that Lim already determines, so that the second image captures additional context sufficient for the image improvement that is Lim's stated objective. This is a combination of Lim's known shadow quantification with Lim's known adjustable capture settings according to their established functions to yield the predictable result that the second image is matched to the detected shadow (MPEP 2143(A)); the configuration value controls what additional context the second image captures. MPEP 2144.05(II).
Claims 2 and 14.
Lim teaches wherein the at least one property of the at least one shadow comprises at least one of a shadow intensity, a shadow complexity, shadow area, or a shadow type (Lim:¶ 124. The device determines the shadow area of the first image.). Claim 14 recites the multi-camera
device counterpart of claim 2 and is rejected for the same reasons, mutatis mutandis.
Claims 4 and 16.
Lim teaches wherein the second camera comprises at least one of a wide angle camera, a
telephoto camera, an optical zoom camera, a standard camera, or an ultra-wide camera (Lim:
¶ 127, 51. The second camera module 344 is a telephoto camera; the lens assembly
properties include optical zoom and a telephoto lens.). Claim 16 recites the multi-camera
device counterpart of claim 4 and is rejected for the same reasons, mutatis mutandis.
Claims 5 and 17.
Lim teaches wherein the at least one additional context comprises at least one of lighting
conditions, object colors, finer details, or textures of the input scene (Lim: ¶ 127. The
telephoto second camera captures subject detail not obtained by the wide-angle first
camera i.e. greater magnification yields finer details). Claim 17 recites the multi-camera device counterpart of claim 5 and is rejected for the same reasons, mutatis mutandis.
Claims 6 and 18.
Lim teaches a system user interface and a display manager that output graphical user interface
elements on the display (Lim: ¶ 96, 104. The system UI 515 configures GUI screens and
the display manager 526 manages output of GUI elements.), but does not state providing an
icon for the removing the at least one shadow based on a shadow removing event.
The Examiner takes Official Notice that providing an icon as a graphical user interface control to invoke a function and indicate its status is well known and conventional (MPEP 2144.03). It would have been obvious before the effective filing date of the claimed invention to provide such an icon among Lim's graphical user interface elements to invoke and indicate the shadow removal function. The motivation would have been to give the user a direct control to trigger shadow removal and to convey that the function is active, a predictable use of a known graphical user interface element. (Claim 18 is additionally rejected under 35 U.S.C. 112(b) above.)
Claims 7 and 19.
Lim teaches wherein the at least one configuration of the second camera comprises at least one
of an optical zoom level or an exposure time (Lim: ¶ 51, 54. Lens properties include optical zoom; the image signal processor 260 controls exposure time of the image sensor.).
Claim 19 recites the multi-camera device counterpart of claim 7 and is rejected for the same
reasons, mutatis mutandis.
Claim 8.
Lim teaches removing the at least one shadow from the first image, based on the at least one of
the at least one property of the at least one shadow or the ROI obtained from the first image, or the at least one additional context obtained from the second image (Lim: ¶ 124, 134-135.
The shadow region determined from the first image and the synthesized telephoto second image content are used to produce the shadow-free composite.).
Claim 10.
As best understood in view of the rejection of claim 10 under 35 U.S.C. 112(b) above, Lim
teaches optical zoom as a lens property and switching to the telephoto second camera (Lim: ¶
51, 158). It would have been obvious to select the optical zoom level from a zoom-in or a zoom-out, as a routine selection among Lim's disclosed optical zoom settings, to frame the additional capture of the shadow region.
Claim 13.
Lim teaches a multi-camera device comprising: a first camera and a second camera (Lim: ¶
120. Camera module 340 includes a first camera module 342 (wide-angle) and a second
camera module 344 (telephoto).); memory storing instructions (Lim: ¶ 118. Memory
330 stores instructions that are executed by the processor.); and at least one processor
operatively connected to the memory, the first camera, and the second camera, wherein the at
least one processor is configured to execute the instructions to (Lim: ¶ 64, 119. Processor
310 is connected to and controls memory 330 and camera module 340.): receive a first image
of an input scene from the first camera (Lim: ¶ 120-122); identify at least one shadow in the
first image (Lim: ¶ 123-124, 146-148); determine at least one of at least one property of the
at least one shadow or a region of interest (ROI) of the at least one shadow in the first image
(Lim: ¶ 124, 147-148); apply at least one configuration to the second camera for obtaining a
second image of the input scene based on the at least one of the at least one property of the at
least one shadow or the ROI of the at least one shadow, to identify at least one additional context of the input scene, where the at least one additional context is not obtained using the first camera (Lim: ¶ 126-127, 158); and remove the at least one shadow from the first image, based on the first image and the second image (Lim: ¶ 134-135). The configuration-selection limitation is rejected for the reasons given for claim 1, mutatis mutandis.
Claim 20.
Lim teaches a method for removing at least one shadow from an image, comprising: capturing,
by a first camera of a multi-camera device, a first image of an input scene (Lim: ¶ 120-122);
identifying, by at least one processor of the multi-camera device, at least one shadow in the first image (Lim: ¶ 123-124); determining, by the at least one processor of the multi-camera
device, an area of the first image covered by the at least one shadow (Lim: ¶ 124. The device
determines the shadow area of the first image.); capturing, by a second camera the multicamera device, a second image of a portion of the input scene comprising the area of the first image covered by the at least one shadow, the second camera having a larger focal length and using a longer exposure time than the first camera (Lim: ¶ 127, 51. The telephoto second
camera, having a larger focal length than the wide-angle first camera, captures the second
image of the subject.); and overlaying, by the at least one processor of the multi-camera device, the second image on the first image to remove the at least one shadow from the first image (Lim: ¶ 134-135. The first and second images are matched and synthesized into a composite from which the shadow is removed.).
Lim does not state using a longer exposure time than the first camera. Lim teaches that the image signal processor controls the exposure time of the camera (Lim: ¶ 54.). The Examiner takes Official Notice that capturing a darker region of a scene with a longer exposure time, to admit more light and recover color and texture detail in that region, is well known and conventional in photography (MPEP 2144.03). It would have been obvious before the effective filing date of the claimed invention to capture the second image of the shadow area, which is darker than the surrounding scene, using a longer exposure time than the first camera, in order to recover color and texture detail in the shadow region for the shadow removal. This is a predictable use of known exposure control directed to the shadow region, notwithstanding that a telephoto capture might in other contexts use a shorter exposure, because here the exposure is selected for the dark shadow area rather than for the framing of a distant subject.
Claims 3, 9, 11, 12 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Lim as applied to claims 1 and 6 above, and further in view of Wang et al., "Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal," (hereinafter “Wang”).
Claims 3 and 15.
Lim teaches determining the region of interest of the at least one shadow (Lim:¶ 124, 147-
148. The device determines the shadow area and a color difference value of the shadow
region.), but does not state that the ROI is a shadow mask which indicates an area of the at least one shadow.
Representing a determined shadow region as a shadow mask is a conventional manner of delineating the shadow area for processing, and is further taught by Wang, whose first generator produces a shadow detection mask indicating the shadow region (Wang:
Abstract.).
It would have been obvious before the effective filing date of the claimed invention
to represent Lim's shadow region as a shadow mask indicating the area of the shadow, in order
to provide a standard, computationally convenient delineation of the shadow region for the
subsequent removal operation, a predictable result.
Claim 9.
As best understood in view of the rejection of claim 9 under 35 U.S.C. 112(b) above, Lim teaches the method of claim 6 and further teaches applying the at least one configuration as
an optical zoom level of the second camera and selecting the optical zoom level of the second
camera for obtaining the second image to identify the additional context (Lim: ¶ 51, 126-127,
158. The lens assembly properties include optical zoom; the telephoto second camera
captures detail of the subject not obtained by the wide-angle first camera.), and analyzing
the property or ROI of the first image and the additional context of the second image and
removing the shadow based on the first image and the second image (Lim:¶ 134-135.).
Selection of the configuration based on the shadow property or ROI is taught for the reasons
given for claim 1.
Lim determines and removes the shadow through color-difference detection, time-of-flight distance measurement, object-coordinate matching, and image synthesis. Lim does not state determining the at least one property of the at least one shadow or the ROI using a first AI
model, or removing the at least one shadow using a second AI model. Wang teaches a shadow framework of two stacked conditional generative adversarial networks, in which a shadow image is fed into a first generator that produces a shadow detection mask, and the shadow image concatenated with the predicted mask is fed into a second generator that recovers a shadow-free image (Wang: Abstract; see also Fig. 2. "a shadow image is fed into the first generator which produces a shadow detection mask. That shadow image, concatenated with its predicted mask, goes through the second generator in order to recover its shadow-free image consequently."). Wang's first generator is a first AI model that determines the ROI of the shadow as the shadow detection mask, and Wang's second generator is a second AI model that removes the shadow.
It would have been obvious before the effective filing date of the claimed invention to implement the shadow detection and the shadow removal of Lim using the first AI model and the second AI model of Wang, as a substitution of one known shadow detection and removal technique for another known technique to obtain the predictable result of detecting and removing the shadow (MPEP 2143 (B)). Lim's multi-camera capture of the first image and the second image is retained and supplies the input to the models, so that the first AI model determines the shadow property or ROI from the first image and the second AI model removes the shadow based on the first image and the additional context of the second image; Lim's principle of operation of capturing complementary images to improve the shadow region is therefore unchanged. A person of ordinary skill would have had a reasonable expectation of success because Wang demonstrates that the two stacked models successfully detect and remove shadows. The motivation would have been to obtain the improved detection and removal accuracy that Wang teaches results from jointly learning the two tasks (Wang: Abstract.).
Claim 11.
As best understood in view of the rejection of claim 11 under 35 U.S.C. 112(b) above, claim 11 recites applying the at least one configuration as an exposure time rather than an optical zoom level. Lim teaches exposure time control of the camera and varying the exposure time of the second camera to obtain the additional context (Lim: ¶¶ 54, 158. The image signal processor 260 controls exposure time.). The first AI model and the second AI model are taught by Wang (Wang: Abstract and Fig. 2.) as set forth for claim 9, and the same rationale and motivation to combine apply, mutatis mutandis.
Claim 12.
Lim in view of Wang teaches the method of claim 11. Lim teaches controlling and varying the exposure time (Lim: ¶ 54.). It would have been obvious that the varying of the exposure time comprises adjusting the time range from a lower to higher range or a higher to lower range, these being the two available directions of any exposure-time adjustment, in order to admit more or less light to recover detail in the shadow region, a predictable result.
Conclusion
The following references are made of record and considered pertinent to applicant's disclosure but are not relied upon. US 2018/0160046 A1 and US 2017/0230585 A1 each disclose a multi-camera device that combines images from asymmetric wide-angle and telephoto cameras using spatial and photometric transforms..
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ross Varndell whose telephone number is (571)270-1922. The examiner can normally be reached M-F, 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’Neal Mistry can be reached at (313)446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Ross Varndell/Primary Examiner, Art Unit 2674
Prosecution Insights