DETAILED ACTION
[1] Remarks
I. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
II. Claims 1-15 are pending and have been examined, where claims 1, 4, 6, 9-10 and 14 is/are rejected, claim 2-3, 5, 7-8, 11-13 and 15 is/are objected to. Explanations will be provided below.
III. Inventor and/or assignee search were performed and determined no double patenting rejection(s) is/are necessary.
IV. Patent eligibility (updated in 2019) shown by the following: Claims 1-15 pass patent eligibility test because there is/are no limitation or a combination of limitations amounting to an abstract idea. Also, the following limitation or the combinations of the limitations: “identifying, by the electronic device, one or more regions to be enhanced in at least one
obtained image frame of the plurality of obtained image frames, based on the at least one
identified motion characteristic associated with the plurality of estimated key points and the at
least one detected action; and generating, by the electronic device, an enhanced image by enhancing the identified one or more regions” effects a transformation or a reduction of a particular article to a different state or thing / adds a specific limitation(s) other than what is well-understood, routine and conventional in the field, or adding unconventional steps that confine the claim to a particular useful application and providing improvements to the technical field of image enhancement, which recite additional elements that integrate the judicial exception into a practical application and amounting significant more.
[2] Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
Use of the word “means” (or “step for”) in a claim with functional language creates a rebuttable presumption that the claim element is to be treated in accordance with 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph). The presumption that 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph) is invoked is rebutted when the function is recited with sufficient structure, material, or acts within the claim itself to entirely perform the recited function. Absence of the word “means” (or “step for”) in a claim creates a rebuttable presumption that the claim element is not to be treated in accordance with 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph). The presumption that 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph) is not invoked is rebutted when the claim element recites function but fails to recite sufficiently definite structure, material or acts to perform that function.
Claim elements in this application that use the word “means” (or “step for”) are presumed to invoke 35 U.S.C. 112(f) except as otherwise indicated in an Office action. Similarly, claim elements that do not use the word “means” (or “step for”) are presumed not to invoke 35 U.S.C. 112(f) except as otherwise indicated in an Office action.
Claim(s) 11-15 are not interpreted under 35 U.S.C. 112(f) or pre-AIA U.S.C. 112 6th paragraph because of the following reason(s): limitations are modified by sufficient structure or material for performing the claimed function.
Claim(s) 1-10 do not require 35 U.S.C. 112(f) or pre-AIA U.S.C. 112 6th paragraph interpretation because they are method claims and / or they are CRM claims.
Upon examination of the specification and claims, the examiner has determined, under the best understanding of the scope of the claim(s), rejection(s) under 35 U.S.C. 112(a)/(b) is not necessitated because of the following reasons: sufficient support are provided in the written description / drawings of the invention.
[3] Grounds of Rejection
Claim Rejections - 35 USC § 103
1. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
2. Claims 1 and 9-10 is/are rejected under 35 U.S.C. 103 as being unpatentable over TUSCH (US 20210279475) in view of Swamy (US 20130328834).
Regarding claim 1, TUSCH discloses a method for motion-based image enhancement, the method comprising:
obtaining, by an electronic device, a plurality of image frames comprising at least one
subject that performs at least one action (see paragraph 710 and figure 7, most common user interface between homeowner and smart home systems is via the web and using a web browser either on a mobile phone, where the mobile phone is read as the electronic device, see figure 4, is the frames captured);
estimating, by the electronic device, a plurality of key points associated with the at least
one subject in the plurality of obtained image frames (see figure 16, plurality of fiducial points are shown);
PNG
media_image1.png
283
623
media_image1.png
Greyscale
detecting, by the electronic device, the at least one action performed by the at least one
subject using the plurality of estimated key points (see figure 14 illustration below);
PNG
media_image2.png
228
505
media_image2.png
Greyscale
identifying, by the electronic device, at least one motion characteristic associated with
the plurality of estimated region of interests’
PNG
media_image3.png
327
582
media_image3.png
Greyscale
identifying, by the electronic device, one or more regions to be enhanced in at least one
obtained image frame of the plurality of obtained image frames (see figure 12, plurality of areas are identified),
generating, by the electronic device, an enhanced image by enhancing the identified one or more regions (see paragraph 1010, spatio-temporal filtering and clustering is applied to group multiple detections of the same object at slightly different offsets and scales, and to improve tracking stability, where the filtering and clustering are the enhancing techniques).
TUSCH is silent in disclosing identifying, by the electronic device, at least one motion characteristic associated with the plurality of estimated key points; and identifying, by the electronic device, one or more regions to be enhanced in at least one obtained image frame of the plurality of obtained image frames, based on the at least one identified motion characteristic associated with the plurality of estimated key points and the at least one detected action.
Swamy discloses identifying, by the electronic device, at least one motion characteristic associated with the plurality of estimated key points (see figure 4, P1 to P8 are estimated key points); and identifying, by the electronic device, one or more regions to be enhanced in at least one obtained image frame of the plurality of obtained image frames, based on the at least one identified motion characteristic associated with the plurality of estimated key points and the at least one detected action (see figure 5, 108 extracting additional gesture point locations from additional captured image frame and the gesture points are employed to new gesture, 120, paragraph 29, FIG. 1 may be used to perform edge detection and/or other image enhancement operations on FRAME 1. The edge detection operations may be used to identify gesture point locations such as end portions P1 and P2 of segment 51).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to include identifying motion characteristics of estimated keypoints because spatial positions such as the hands/joints, of the motion characteristics provide the temporal context, as in how they are moving, required to distinguish between different actions improving gesture recognition.
Regarding claim 9, Swamy discloses the method of claim 1, wherein the identifying the one or more regions to be enhanced comprises:
comparing, by the electronic device, at least one computed value of the at least one identified motion characteristic associated with the plurality of estimated key points with at least one expected value of the at least one identified motion characteristic associated with the plurality of estimated key points (see figure 5, 116 compared stored coordinates to template coordinates);
determining, by the electronic device, a deviation of the at least one computed value corresponding to each of the plurality of estimated key points from the at least one expected value (see figure 5, 114 determine time since last new gesture point location, where the time is the value computed); and
determining, by the electronic device, a first set of key points of the plurality of estimated key points having the deviation greater than a threshold value (see figure 5, character threshold is read as threshold value “TIME > CHARACTER THRESHOLD”, this deviation could be spatial or temporal).
Regarding claim 10, see the rationale and rejection for claim 1. In addition, see paragraph 323, processor is employed, which requires instructions stored in memory.
3. Claims 4, 6 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over TUSCH (US 20210279475) in view of Swamy (US 20130328834) and Wong (US 11094039).
Regarding claim 4, the combination of TUSCH and Swamy as a whole discloses all the limitations of claim 1 but is silent in disclosing the method of claim 1, wherein the generating the enhanced image comprises generating at least one of a High Dynamic Range (HDR) image, a de-noised image, a blur corrected image, or a reflection removed image.
Wong discloses the method of claim 1, wherein the generating the enhanced image comprises generating at least one of a High Dynamic Range image (see column 25, lines 1-5, which may represent, e.g., a High Dynamic Range capture embodiment, involving the fusion of an underexposed EV−1 image, a correctly-exposed image, and an over-exposed EV+2 image), a de-noised image (see column 19, lines , a blur corrected image, or a reflection removed image.
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to include High Dynamic Range image in order to HDR imaging expands the contrast ratio of the image allowing the final image to preserve details in highly illuminated areas and underexposed areas.
Regarding claim 6, Wong discloses the method of claim 4, wherein the generating the de-noised image comprises generating a motion map based on the at least one identified motion characteristic associated with the plurality of estimated key points (see column 19, lines 29-31, the fused image 174 may have noise reduction 176 applied to it according to the fusion-adaptive noise map 172, resulting in fused, adaptively denoised image 178, figure 1C include points XR, X1, … XN). See the motivation for claim 4. In addition, denoising an image reveals hidden details such as fine textures, sharp edges, and gradients. Removing this noise allows the the image to become visible.
Regarding claim 14 see the rationale and rejection for claim 4.
[4] Claim Objections
Claim(s) 2-3, 5, 7-8, 11-13 and 15 is/are objected to as being dependent upon a rejected base claim but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
With regards to claim 2, the examiner cannot find any applicable prior art providing teachings for the following limitation(s): the method of claim 1, wherein the identifying the one or more regions to be enhanced comprises: determining, by the electronic device, an optimal motion map using a plurality of optimal image frames based on at least one predicted local motion region and the plurality of estimated key points; performing, by the electronic device, localization of a spatial-temporal artifact for the plurality of optimal image frames based on the determined optimal motion map, the at least one identified motion characteristic associated with the plurality of estimated key points, and the at least one detected action; and identifying, by the electronic device, the one or more regions to be enhanced in the at least one obtained image frame of the plurality of obtained image frames based on the localization of the spatial-temporal artefact, wherein the one or more regions comprise at least one image artifact; in combination with the rest of the limitations of claim 1.
Swamy discloses the method of claim 1, wherein the identifying the one or more regions to be enhanced comprises:
determining, by the electronic device, an
PARK (US 20170147077) discloses
performing, by the electronic device, localization of a spatial-temporal artifact for the plurality of signal
TUSCH discloses identifying, by the electronic device, the one or more regions to be enhanced in the at least one obtained image frame of the plurality of obtained image frames based on the spatio-temporal filtering and clustering is applied to group multiple detections of the same object at slightly different offsets and scales, and to improve tracking stability, where the filtering and clustering are the enhancing techniques).
Regarding claim 12, see the rationale and objection for claim 2.
Claim(s) 3 is/are objected as well because it is dependent on a claim with allowable subject matter.
With regards to claim 5, the examiner cannot find any applicable prior art providing teachings for the following limitation(s): “the method of claim 4, wherein the generating the HDR image comprises: clustering, by the electronic device, the identified one or more regions in the at least one obtained image frame and clustering the plurality of obtained image frames into a plurality of frame groups, respectively, based on the at least one identified motion characteristic associated with the plurality of estimated key points and the at least one detected action, wherein the plurality of frame groups comprise a first frame group including frames having a lower displacement, a second frame group including frames having a medium displacement, and a third frame group including frames having a higher displacement; generating, by the electronic device, a high exposure frame using the frames in the first frame group; generating, by the electronic device, a medium exposure frame using the frames in the second frame group; generating, by the electronic device, a low exposure frame using the frames in the third frame group; and blending, by the electronic device, the generated high exposure frame, the generated medium exposure frame, and the generated low exposure frame to generate the HDR image” in combination with the rest of the limitations of claim 1.
Swamy discloses the method of claim 4, wherein the generating the HDR image comprises:
clustering, by the electronic device, the identified one or more regions in the at least one obtained image frame and clustering the plurality of obtained image frames into a plurality of frame groups, respectively, based on the at least one identified motion characteristic associated with the plurality of estimated key points and the at least one detected action (see figure 4,key points P1 and P2 are clustered together), wherein the plurality of frame groups comprise
a first frame group including frames having a lower displacement, a second frame group including frames having a lower lower displacement (see figure 4, P1-P2, P3-P4, P5-P6 and P7-P8 all have the same displacements).
Swamy is silent in disclosing generating, by the electronic device, a high exposure frame using the frames in the first frame group; generating, by the electronic device, a medium exposure frame using the frames in the second frame group; generating, by the electronic device, a low exposure frame using the frames in the third frame group; and blending, by the electronic device, the generated high exposure frame, the generated medium exposure frame, and the generated low exposure frame to generate the HDR image.
Regarding claim 15 see the rationale and objection for claim 5.
Regarding claim 7, the examiner cannot find any applicable prior art providing teachings for the following limitation(s): the method of claim 4, wherein the generating the blur corrected image comprises: determining, by the electronic device, whether each of the at least one identified motion characteristic exceeds a pre-defined threshold; and generating, by the electronic device, the blur corrected image by applying a blur correction to one or more regions surrounding at least one key point of which a motion characteristic exceeds the pre-defined threshold; in combination with the rest of the limitations of claims 1 and 4.
Swamy discloses determined time compared to one or more thresholds such as a character threshold, predetermined amount of time that is typical for entering a single character, and/or a clear threshold, where a predetermined amount of time at which the memory is reset for entering a new character (see paragraph 45), but is silent in disclosing determining, by the electronic device, whether each of the at least one identified motion characteristic exceeds a pre-defined threshold; and generating, by the electronic device, the blur corrected image by applying a blur correction to one or more regions surrounding at least one key point of which a motion characteristic exceeds the pre-defined threshold.
Regarding claim 8, the examiner cannot find any applicable prior art providing teachings for the following limitation(s): the method of claim 4, wherein the generating the reflection removed image comprises: determining, by the electronic device, a correlation between at least one identified motion characteristic associated with the plurality of estimated key points of a first subject with at least one identified motion characteristic associated with the plurality of estimated key points of a second subject; classifying, by the electronic device, at least one highly correlated key point of the second subject as a reflection key point; generating, by the electronic device, a reflection map using the classified at least one highly correlated key point; and generating, by the electronic device, the reflection removed image using the generated reflection map; in combination with the rest of the limitations of claims 1 and 4.
Ryu (US 20120172867) discloses a motion map comprise a local motion map reflecting the movement of one or more position sensors 40, 42, 46 in a region of heart 12 or a global motion map reflected in (see paragraph 29), but is silent in disclosing the method of claim 4, wherein the generating the reflection removed image comprises: determining, by the electronic device, a correlation between at least one identified motion characteristic associated with the plurality of estimated key points of a first subject with at least one identified motion characteristic associated with the plurality of estimated key points of a second subject; classifying, by the electronic device, at least one highly correlated key point of the second subject as a reflection key point; generating, by the electronic device, a reflection map using the classified at least one highly correlated key point; and generating, by the electronic device, the reflection removed image using the generated reflection map.
Regarding claim 11, the examiner cannot find any applicable prior art providing teachings for the following limitation(s): the electronic device of claim 10, wherein the image processing controller is further configured to: determine a pose of a subject in a scene being captured; identify the plurality of key points from the determined pose; measure a plurality of motion parameters for the plurality of key points, respectively; determine whether each of the plurality of measured motion parameters exceeds a pre- defined threshold; and apply a blur correction to regions surrounding at least one key point of which a measured motion parameter exceeds the pre-defined threshold; in combination with the rest of the limitations of claim 10.
TUSCH discloses determine a pose of a subject in a scene being captured (see figure 12, the hands are being recognized).
Swary discloses identify the plurality of key points from the determined pose (see figure 4, P1-P8 are key points); measure a plurality of coordinates
GUPTA (US 20190026869) discloses apply a blur correction to regions surrounding at least one key point paragraph 32, the decoder of the target was used to generate the clean deblurred patch, once all the individual patches of the testing image are deblurred).
Regarding claim 13, the examiner cannot find any applicable prior art providing teachings for the following limitation(s): the electronic device of claim 12, wherein the image processing controller is further configured to: determine the plurality of optimal image frames from the plurality of obtained image frames based on the at least one detected action; predict the at least one local motion region in at least one optimal image frame of the determined plurality of optimal image frames based on the at least one detected action; determine a digital skeleton using the plurality of estimated key points; and determine the optimal motion map using the plurality of optimal image frames based on the at least one predicted local motion region and the digital skeleton; in combination with all the limitations of claims 10 and 12.
Kwon (US 20220066544) discloses the electronic device of claim 12, wherein the image processing controller is further configured to:
determine the
predict the at least one local motion region in at least one
determine a digital skeleton using the plurality of estimated key points (see figure 2B, above 222a keypoints are shown); and
determine the optimal motion map using the plurality of
PNG
media_image4.png
356
743
media_image4.png
Greyscale
.
CONTACT INFORMATION
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALEX LIEW (duty station is located in New York City) whose telephone number is (571)272-8623 (FAX 571-273-8623), cell (917)763-1192 or email alexa.liew@uspto.gov. Please note the examiner cannot reply through email unless an internet communication authorization is provided by the applicant. The examiner can be reached anytime.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MISTRY ONEAL R, can be reached on (313)446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALEX KOK S LIEW/Primary Examiner, Art Unit 2674 Telephone: 571-272-8623
Date: 9/11/26
Prosecution Insights