Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Allowable Subject Matter
Claim 6-7, 10 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 8-9, 11-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Qiu U.S. Patent/PG Publication 20230345133 in view of Shen U.S. Patent/PG Publication 20170294000.
Regarding claim 1 (independent):
An electronic device comprising: a camera module including a camera; a display module including a display; memory storing instructions and at least one processor, comprising processing circuitry, operatively connected to the camera module and the display module, wherein the instructions, when executed by the at least one processor, individually or collectively, cause the electronic device to: (Qiu [0125] The electronic device 30 includes but is not limited to components such as a radio frequency unit 301, a network module 302, an audio output unit 303, an input unit 304, a sensor 305, a display unit 306, a user input unit 307, an interface unit 308, a memory 309, and a processor 310.)(Qiu [0029] In this embodiment of this application, the shooting method is applied to an electronic device that has a display screen and a camera, such as a mobile phone, a tablet computer, or a camera.)
identify one or more objects in a first image, (Qiu [0031] In step S100, the first starry sky image is a starry sky image acquired by a camera of the electronic device. In actual application, the first starry sky image may be a preview image formed after a to-be-shot star area enters a field of view of the camera in a case that the camera is enabled. [0032] Step S200: Determine a first planet in the first starry sky image according to a geographical location of the electronic device and/or recognition of the first starry sky image.).
based on pixel values of regions respectively corresponding to the one or more identified objects, determine at least one map related to a pixel value of a first region of the first image, (Qiu [0033] In step S200. to subsequently adjust the brightness of each planet in a shot image according to magnitude brightness, a first planet included in a to-be-shot star region need to be determined. [0034] Because different star regions can be viewed at different locations on the earth, when the electronic device shoots an image, the geographical location of the electronic device can be recorded, and then each first planet in the shot starry sky image can be found and determined according to the recorded geographical location. In some embodiments, at a same location on the earth, star regions shot by the camera at different shooting angles are also different. Therefore, when the electronic device shoots a starry sky image, a current location of the electronic device and a current shooting angle may be determined by means of positioning, and then each planet included in the first starry sky image acquired through shooting is determined according to the current location and the current shooting angle.)(Qiu [0085] In an implementation, the planet information further includes a type of the planet, a distance from the earth, a magnitude, current coordinates in the sky, and the like. The information may be presented when the user taps the target first planet after the user enables a star map knowledge pattern, so that corresponding planet information can be more comprehensively propagated to the user.).
based on at least one of metadata of the first image (Qiu [0041] where the target time is time at which the first starry sky image is acquired.)(Qiu [0046] Step S101: Acquire a target rotation angle according to target duration, where the target rotation angle is an angle of rotation of the earth within the target duration, and the target duration is duration from receiving of a shooting input by the electronic device to imaging.)(Qiu [0032] Step S200: Determine a first planet in the first starry sky image according to a geographical location of the electronic device and/or recognition of the first starry sky image.), an input for determining a composition of the second image, (Qiu [0007] acquiring a first starry sky image) (Qiu [0041] Step S400: Adjust brightness of the first planet in the first starry sky image according to the target time and the magnitude brightness, where the target time is time at which the first starry sky image is acquired.)
based on the at least one determined map, change a pixel value of at least partial area of the (Qiu [0042] In step S400, because the magnitude brightness of the planet is a fixed value, and actual brightness presented by the planet at different times is different, display brightness of the first planet in a shot image may be simulated by using a synthesis algorithm after the magnitude brightness of the first planet is calculated. The display brightness shows real brightness of the first planet when the first starry sky image is acquired, and corresponding brightness of the first planet is rendered according to the display brightness, so that real brightness of the first planet is clearly presented on the first starry sky image. Brightness of each first planet in the first starry sky image is visual performance of true brightness of the first starry sky image, the first starry sky image can have a real and clear planet visual effect.).
Qiu does not teach a second image. In a related field of endeavor, Shen teaches:
based on pixel values of regions respectively corresponding to the one or more identified objects, determine at least one map related to a pixel value of a first region of the first image, (Shen [0038] In various embodiments, the process begins at block 310, where the composition of a target image is to be determined, e.g., by composition detector 210 of FIG. 2. In various embodiments, composition of the target image relates to the arrangement of visual objects on the image, e.g., the layout and the disposition of foreground objects. Objects on the target image are often the output of object recognition or scene parsing. In some embodiments, objects on an image are to be recognized based on an object model, e.g., template matching, constellations, bags of features, or shape models, etc. Composition detector 210 uses an efficient and reliable scene parsing system to recognize different objects on the target image and further their unique arrangements on the target image. In some embodiments, a CNN is utilized for sky segmentation and object recognition. The same or a different CNN is used to generate a feature vector to represent the composition of the target image.).
based on at least one of metadata of the first image an input for determining (Shen [0049] In various embodiments, process 400 begins at block 410, where respective histograms associated with the scene parsing labels of a spatial grid of a training image is to be computed, e.g., by composition detector 210 of FIG. 2. The training image is divided into a plurality of identifiable cells in a spatial grid.), a composition of the second image (Shen [0017] In at least some embodiments of the present disclosure, reference images for sky replacement are selected based on their image composition, such as image layout and/or image content. Image layout generally refers to a layout of an image, such as the arrangement of the sky and the foreground (also referred to as the non-sky part/components in this disclosure), the placement of objects (e.g., human subjects) in the foreground, etc. I), or a (Shen [0039] At block 320, a reference image sharing similar composition with the target image is to be identified, e.g., based on the feature vector of the target image, e.g., by sky search engine 220 of FIG. 2. In various embodiments, composition of respective images in an image bank is to be identified using similar techniques for identifying the composition of the target image. Therefore, sky search engine 220 is capable to search appropriate images for sky replacement, e.g., by comparing the composition of the target image and the composition of a candidate image.).
based on the at least one determined map, change a pixel value of at least partial area of the second image that corresponds to the first region, obtain a third image by synthesizing, with a second region of the first image, the second image in which the pixel value is changed and display, via the display module, the third image (Shen [0041] Next, at block 330, a new image is to be generated, e.g., by replacing the sky in the target image with the sky in a selected reference image, e.g., by sky editor 230 of FIG. 2. Sky replacement, as used in this disclosure, refers to replacing the content of the sky. In various embodiments, sky replacement means to replace the whole background of an image.)
Therefore, it would have been obvious before the effective filing date of the claimed invention to use a second image as taught by Shen. The rationale for doing so would have been that it combines prior art elements according to known methods to yield predictable results where Qiu takes a photo, identifies objects, and generates a composited enhanced image and Shen takes a photo, identifies objects, and generates a composited enhanced image, where Shen is merely using an image and Shen is using data, and the input and end result for both are the same. Therefore it would have been obvious to combine Shen with Qiu to obtain the invention.
Regarding claim 2:
The electronic device of claim 1, has all of its limitations taught by Qiu in view of Shen. Qiu further teaches wherein the instructions, when executed by at least one processor, individually or collectively, cause the electronic device to:
input the first image into a previously trained image classification model, based on an output of the image classification model, identify whether the first image includes at least one predetermined object, and based on identifying that the first image includes the at least one predetermined object, identify one or more objects in the first image (Qiu [0032] Step S200: Determine a first planet in the first starry sky image according to a geographical location of the electronic device and/or recognition of the first starry sky image.)(Qiu [0121] In this embodiment of this application, when the first acquiring module 21 acquires the first starry sky image, the first determining module 22 determines the first planet in the first starry sky image according to the geographical location of the electronic device and/or recognition of the first starry sky image.).
Regarding claim 3:
The electronic device of claim 1, has all of its limitations taught by Qiu in view of Shen. Shen further teaches wherein the instructions, when executed by at least one processor, individually or collectively, cause the electronic device to:
based on a pixel value of the first image, obtain an entire brightness value of the first image, and based on the obtained entire brightness value of the first image, change an entire brightness value of the second image (Shen [0046] As a result, the sky editing system presents a new image with a user preferred sky, which is to be appreciated by the user as natural and holistic, by harmonizing the foreground of the new image with the replaced sky.)(Shen [0069] At block 710, the luminance of the non-sky part of the target image is adjusted based on the reference image, e.g., by sky editor 230 of FIG. 2. In some embodiments, the luminance channel in the Lab color space and a weighted luminance from the foreground of the reference image are used.).
There is a prima facie case of obviousness since the limitation is directed to common practices which the court has held normally require only ordinary skill in the art and hence are considered routine expedients are discussed below. See MPEP 2144.04. Rearrangement of Parts “the court held that mere duplication of parts has no patentable significance unless a new and unexpected result is produced.” The motivation to modify is that the image is being harmonized (Shen [0046]) which can be the first image being modified to match the second, or the second being modified to match the first, where both create harmonization.
Therefore, it would have been obvious before the effective filing date of the claimed invention to adjust as taught by Shen. The motivation for doing so would have been to harmonize the image (Shen [0046]). Therefore it would have been obvious to combine Shen with Qiu to obtain the invention.
Regarding claim 4:
The electronic device of claim 1, has all of its limitations taught by Qiu in view of Shen. Shen further teaches wherein the instructions, when executed by at least one processor, individually or collectively, cause the electronic device to:
based on a pixel value of the second image, obtain a value map corresponding to the second image, and based on a lightness value of the at least one map related to the pixel value of the first region, adjust a pixel value included in the value map (Shen [0046] As a result, the sky editing system presents a new image with a user preferred sky, which is to be appreciated by the user as natural and holistic, by harmonizing the foreground of the new image with the replaced sky.)(Shen [0070] At block 720, the color temperature of the non-sky part of the target image is adjusted based on the reference image, e.g., by sky editor 230 of FIG. 2. Since the foreground appearance between the target and reference image is different in many cases, instead of transferring color statistics, sky editor 230 transfers the tone by computing the color temperature in the XYZ color space. In some embodiments, sky editor 230 uses a smooth transfer function for histogram matching with 32 bins as illustrated in Eq. 3.)
Therefore, it would have been obvious before the effective filing date of the claimed invention to adjust as taught by Shen. The motivation for doing so would have been to harmonize the image (Shen [0046]). Therefore it would have been obvious to combine Shen with Qiu to obtain the invention.
Regarding claim 5:
The electronic device of claim 1, has all of its limitations taught by Qiu in view of Shen. Shen further teaches wherein the instructions, when executed by at least one processor, individually or collectively, cause the electronic device to:
identify whether the one or more identified objects include at least one predetermined object, based on identifying that the one or more identified objects include the at least one predetermined object, determine at least one region corresponding to the at least one predetermined object, and generate a mask image by obtaining a pixel value of the at least one determined region (Shen [0043] To replace the sky, the maximum rectangular sky region in the sky mask is to be extracted first. Then, the sky editor 230 rescales the extracted sky region to the size of the minimum rectangle that covers all the sky region of the target image. When generating the new image, at least in some embodiments, sky editor 230 also adjusts characteristics of the foreground objects in the new image, e.g., the luminance, the color temperature, the saturation, etc., according to the similar characteristics in the reference image.).
Regarding claim 8:
The electronic device of claim 1, has all of its limitations taught by Qiu in view of Shen. Qiu further teaches wherein the instructions, when executed by at least one processor, individually or collectively, cause the electronic device to:
detect an input to one of the plurality of objects, (Qiu [0080] Step S601: Receive a second input for the first planet. [0081] In step S601, the second input is a selection input of the user for a location corresponding to the first planet in the first starry sky image, and may be a touch, tap, or focus operation on a screen area in which the first planet in the first starry sky image is located.)
and based on the detected input, adjust the pixel value of the second image (Qiu [0082] Step S602: Display, in response to the second input, planet information corresponding to the first planet.) since pixel values are modified to show information.
Regarding claim 9:
The electronic device of claim 1, has all of its limitations taught by Qiu in view of Shen. Qiu further teaches wherein the instructions, when executed by at least one processor, individually or collectively, cause the electronic device to:
based on metadata of an initial input image obtained by the camera module, determine an initial composition of a celestial object image corresponding to the initial input image, obtain an initial synthesis frame by synthesizing the celestial object image generated based on the determined initial composition with the initial input image, based on a time stamp of each of one or more input images further obtained by the camera module, adjust the composition of the celestial object image, obtain one or more synthesis frames by synthesizing, with each of the one or more input images, respective celestial object image of which the composition is adjusted based on each time stamp, and obtain a synthesis video by sequentially connecting the initial synthesis frame and the obtained synthesis frames (Qiu [0054] In step S112, after the multiple first images are continuously acquired by shooting, the multiple first images are aligned and synthesized into one target image, to acquire the first starry sky image. Because locations of the to-be-shot planet are the same in the first images acquired by shooting for multiple consecutive times, when the multiple first images are subsequently aligned and synthesized into one target image, each first planet images shot are not ambiguous due to displacement.)(Qiu [0058] Step S102: Acquire N starry sky images, where backgrounds of the N starry sky images are the same, and the N starry sky images are acquired by performing shooting at an interval of preset time.)(Qiu [0060] In a process of acquiring the N starry sky images, the preset time and the first rotation angle are acquired, and the first rotation angle is an angle change value required in a shooting direction by two adjacent starry sky images in the N starry sky images that present a star orbit effect of a planet location change. Then, the micro cloud platform is controlled to rotate according to the preset time and the first rotation angle, and each time after the micro cloud platform stop rotating, the camera is controlled to shoot one starry sky image, to continuously perform shooting at an interval of preset time until the N starry sky images are acquired.).
Regarding claim 11 (independent):
The claim is a/an parallel version of claim 1. As such it is rejected under the same teachings.
Regarding claim 12:
The claim is a/an parallel version of claim 2. As such it is rejected under the same teachings.
Regarding claim 13:
The claim is a/an parallel version of claim 3. As such it is rejected under the same teachings.
Regarding claim 14:
The claim is a/an parallel version of claim 4. As such it is rejected under the same teachings.
Regarding claim 15:
The claim is a/an parallel version of claim 5. As such it is rejected under the same teachings.
Regarding claim 16 (independent):
The claim is a/an parallel version of claim 1. As such it is rejected under the same teachings.
Conclusion
For the prior art referenced and the prior art considered pertinent to Applicant’s disclosure but not relied upon, see PTO-892 “Notice of References Cited”.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASON PRINGLE-PARKER whose telephone number is (571) 272-5690 and e-mail is jason.pringle-parker@uspto.gov. The examiner can normally be reached on 8:30am-5:00pm est Monday-Friday. If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, King Poon can be reached on (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, seehttp://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASON A PRINGLE-PARKER/
Primary Examiner, Art Unit 2617
Prosecution Insights