DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 4-9, 11-12 and 14-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al. (US 2010/0173678 A1, cited in IDS) in view of Choe et al. (US 2021/0250510 A1).
Kim et al. discloses an electronic device (Fig.1: mobile terminal 100) comprising:
a camera (Fig.1: camera 121);
a display (Fig.1: display 151);
memory (Fig.1: memory 160), comprising one or more storage media, storing instructions ([0074]: “the memory 160 can store a program for the operation of the controller 180”); and
at least one processor comprising processing circuitry (Fig.1: controller 180),
wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:
obtain shooting information via the camera ([0100] and [0111]: the controller 180 receives and displays the full image detected by the camera image sensor);
identify an object area including a designated object based on the shooting information through a framing engine ([0122]: “In this state, the controller 180 checks whether a face recognition feature has been set (S32). If the user has set the face recognition feature using a particular menu option, for example (Yes in S32), the controller 180 displays composition information related to face recognition regions at a particular position of the main display region (S33).” The composition information related to face recognition regions corresponds to the claimed object area),
display, via the display, a preview image corresponding to the shooting information based on the identified object area (Fig.7: the image displayed at the main display region corresponds to the claimed preview image),
display, via the display, a map view image corresponding to the shooting information (Fig.7: the miniaturized image 40 corresponds to the claimed map view image),
display, via the display, a first object area on the preview image while displaying the preview image (Fig.15: each of the face recognition regions 45 corresponds to the claimed first object area), and
display, via the display, a preview area for the preview image (Fig.7: zoom guide 41 corresponds to the claimed preview area)
Kim et al. fails to disclose displaying a second object area on the map view image while displaying the map view image,
wherein a position of the first object area on the preview image is determined based on the preview image and coordinate information of the object area, and
wherein a position of the second object area on the map view image is determined based on the map view image and the coordinate information of the object area.
However, Choe et al. teaches displaying a second object area on the map view image while displaying the map view image (Fig.2G: the full input image display on the main display area corresponds to the claimed map view image; the candidate zoom previews 224 corresponds to the claimed preview image. The bounding box 230 displayed on the full input image/map view image corresponds to claimed second object area),
wherein a position of the first object area on the preview image is determined based on the preview image and coordinate information of the object area ([0055]: the height and width of salient blob 240 are measured. The center point 242 is determined from the midpoint of the height and width, or from a weighted midpoint. The height, width, midpoint, center point correspond to the claimed coordinate information), and
wherein a position of the second object area on the map view image is determined based on the map view image and the coordinate information of the object area ([0055], [0057]: a center point is determined from the salient object region. The center point frames bounding box 230. Each bounding box 230 pairs with a zoom preview 244).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Kim et al. with the teaching of Choe et al. to display a second object area on the map view image while displaying the map view image, wherein a position of the first object area on the preview image is determined based on the preview image and coordinate information of the object area, and wherein a position of the second object area on the map view image is determined based on the map view image and the coordinate information of the object area, so as to allow the user to identify and track a particular subject, automatically frame that subject in the primary preview, and see the corresponding subject region in the miniaturized contextual view.
As to claim 2, Kim et al. in view of Choe et al. discloses the electronic device of claim 1, wherein, to identify the object area through the framing engine, the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:
obtain an edit image from the shooting information in accordance with a designated aspect ratio (Kim et al.: [0100]: the full image can be considered as an edit image from the image signal output from the sensor in accordance with a designated/default aspect ratio).
The above combination fails to disclose obtaining a framing image for the framing engine based on the edit image; and
identify the object area based on a framing object area in which the designated object is located within the framing image.
However, Choe et al. further teaches obtain a framing image for the framing engine based on the edit image ([0037] and [0053]: providing input image frame 224 to machine-learning algorithm 149, which processes the input image frame to generate salient map 236 identifying regions containing salient information 238. The salient map 236 corresponds to the claimed framing image. The machine-learning object-identification processing corresponds to the claimed framing engine); and identify the object area based on a framing object area in which the designated object is located within the framing image ([0054]: clustering the object-related salient information 238 in the salient map 236 into one or more salient blobs 240. Each salient blob 240 corresponds to a framing object area in which the designated object is located within the framing image).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to further modify the combination Kim et al. and Choe et al. with the teaching of Choe et al. to obtain a framing image for the framing engine based on the edit image; and identify the object area based on a framing object area in which the designated object is located within the framing image, so as to prevent distortion, unintended clipping, or unused display area when the selected region is enlarged to fill the preview display.
As to claim 4, Kim et al. in view of Choe et al. discloses the electronic device of claim 2. The above combination fails to disclose the object area is obtained based on coordinate information of the framing object area in which the designated object is located within the framing image and the edit image.
However, Choe et al. further teaches the object area is obtained based on coordinate information of the framing object area in which the designated object is located within the framing image and the edit image ([0081-0082]: a pixel coordinate of a feature point 250 is identified and a center point is determined using the coordinates. Then the center point of the feature points and the corresponding bounding box 230 are used to crop input image frame 224 and generate the zoomed output).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to further modify the combination Kim et al. and Choe et al. with the teaching of Choe et al. such that the object area is obtained based on coordinate information of the framing object area in which the designated object is located within the framing image and the edit image, so as to enable the controller to accurately identify the object area of the full image corresponding to the object area displayed in the zoomed image.
As to claim 5, Kim et al. in view of Choe et al. discloses the electronic device of claim 2,
wherein the preview image is obtained by reducing or enlarging a portion of the edit image while maintaining the designated aspect ratio (Kim: Fig.7: the lower half of the figure shows that the preview image displayed at the main display region is a zoomed image/enlarged image. It is displayed in the exact same area as the full image in the upper half of the figure, thus, the aspect ratio is unchanged),
wherein the preview image includes the object area (Kim et al.: Fig.7: the bottom half of the figure shows that zoomed image displayed in the main display region include the image content in the zoom position region/zoom guide 41),
wherein the position of the first object area is determined based on the edit image (Choe et al.: [0071]: determining the optimal zoom by comparing the size of bounding box 230 with the dimensions of input image frame 224), a magnification of the preview image (Choe et al.: [0089-0090]: determining a zoom rate from the selected object region’s size relative to input image frame 224 and display the magnified preview), and the coordinate information of the object area (Choe et al.: [0091-0093]: extracting and tracking object feature points and continually update the preview center and zoom rate).
As to claim 6, Kim et al. in view of Choe et al. discloses the electronic device of claim 2,
wherein the map view image is obtained by reducing or enlarging the edit image while maintaining the designated aspect ratio (Kim et al.: [0103]: miniaturized image 40 is a miniaturized version of the full image), and
wherein the position of the second object area is determined based on the edit image, a magnification of the map view image, and the coordinate information of the object area (Choe et al.: [0057]: displaying bounding box 230 in input image frame 224 at the area corresponding to salient blob 240 and zoom preview 244. [0071]: comparing bounding box and input image dimensions to determine magnification).
As to claim 7, Kim et al. in view of Choe et al. discloses the electronic device of claim 1.
The above combination fails to disclose wherein the shooting information is obtained at an application layer, and
wherein the object area is obtained at the application layer.
However, Choe et al. further teaches the shooting information is obtained at an application layer, and wherein the object area is obtained at the application layer ([0034-0037]: program 140 includes an application 147, an application programming interface 145, and a machine-learning algorithm 149. Application 147 supports the camera zoom user interface and that machine-learning algorithm 149 may operate as part of application 147 to process input image frame 224 and identify object regions).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to further modify the combination Kim et al. and Choe et al. with the teaching of Choe et al. such that the shooting information is obtained at an application layer, and wherein the object area is obtained at the application layer, so as to enable the camera application to receive current preview frames, identify target object regions, and use the results directly to control zoomed preview and miniaturized view.
As to claim 8, Kim et al. in view of Choe et al. discloses the electronic device of claim 1, wherein the preview image is obtained in accordance with a zoom-in operation of the camera executed based on identifying the object area (Kim et al.: Figs.17, 19, 21 and 22-23; [0140], [0142], [0145]: face recognition is performed on the full camera image to identify face-recognition regions and in response to selection of a recognized face, shifting the central position of the zoomed preview and adjusting the zoom magnification so that the selected face recognition region is displayed in the preview image).
As to claim 9, Kim et al. in view of Choe et al. discloses the electronic device of claim 1, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:
receive a touch input for selecting another object within the preview image or the map view image (Kim et al.: Fig.12D; [0117]: displaying a miniaturized image 40 containing persons or objects and receiving a circular touch input around one or more persons or objects that the user wants displayed in the zoomed preview);
display, on the display, a new preview image based on the other object area (Kim et al.: Fig.12D; [0117]);
display, via the display, a third object area on the new preview image while displaying the new preview image (Kim et al.: Fig. 17; [0140]: when the user selects an indicator corresponding to a recognized face outside the currently zoomed image, controller 180 shifts the central position of the zoomed image and adjusts the zoom magnification to display the selected face recognition region. The controller displays the selected face recognition region in the new zoomed image); and
display, via the display, a new preview area for the new preview image (Kim et al.: Fig.12A; [0114]: when the user selects another portion of the miniaturized image 40, controller 180 shifts the central position of the zoomed preview, and both the zoomed preview and zoom guide 41 are shifted to the newly selected position) and a fourth object area on a new map view image while displaying the new map view image (Kim et al.: Fig.15; [0129]: displaying recognized faces that have been excluded from the preview using dotted lines and displaying notification information identifying recognized faces located to the left or right of the changed preview).
The above combination fails to disclose
identify another object area corresponding to the other object based on receiving the touch input;
wherein a position of the third object area on the preview image is determined based on the new preview image and coordinate information of the other object area, and
wherein a position of the fourth object area on the map view image is determined based on the map view image and the coordinate information of the other object area.
However, Choe et al. further teaches identify another object area corresponding to the other object based on receiving the touch input (Figs.5A and 5C; [0076]: the optimal blob 520 corresponds to the claimed other object area);
wherein a position of the third object area on the preview image is determined based on the new preview image and coordinate information of the other object area (Fig.2F; [0055]: measuring the height and width of salient blob 240, determining center point 242 based on the midpoint or weighted midpoint of those dimensions, and using center point 242 to frame bounding box 230), and
wherein a position of the fourth object area on the map view image is determined based on the map view image and the coordinate information of the other object area (Fig.2G; [0057]: Choe teaches displaying preview bounding box 230 in input image frame 224 at the area corresponding to salient blob 240 and its associated zoom preview 244, and pairing each bounding box with its corresponding enlarged zoom preview. In other words, Choe teaches using the position and dimensions of the selected object region to display a corresponding object area in the wider input image).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to further modify the combination of Kim et al. and Choe et al. with the teaching of Choe et al. to identify another object area corresponding to the other object based on receiving the touch input; wherein a position of the third object area on the preview image is determined based on the new preview image and coordinate information of the other object area, and wherein a position of the fourth object area on the map view image is determined based on the map view image and the coordinate information of the other object area, so as to accurately position corresponding face or object indicators after the user selected a different object and the preview and miniaturized view were updated.
Method claims 11-12 and 14-18 recite substantially similar subject matter as disclosed in claims 1-2, 4-7 and 9, respectively; therefore, they are rejected for the same reasons.
Claims 19 and 20 recite substantially similar subject matter as disclosed in claims 1 and 9, respectively; therefore, they are rejected for the same reasons.
Claim(s) 3 and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al. (US 2010/0173678 A1, cited in IDS) in view of Choe et al. (US 2021/0250510 A1) as applied to claim 2 above, and further in view of Ubillos et al. (US 2013/0235071 A1).
As to claim 3, Kim et al. in view of Choe et al. discloses the electronic device of claim 2. The above combination fails to disclose the framing image is obtained by reducing or enlarging the edit image in accordance with the designated aspect ratio.
However, Ubillos et al. teaches the framing image is obtained by reducing or enlarging the edit image in accordance with the designated aspect ratio (Figs.18A-18C; [0199]: the cropping tool maintains the aspect ratio of the original image or a predefined aspect ratio selected by the user and stretches the cropped image to match the desired image size. [0211]: the users performs a spreading gesture on the image; in response, the cropping tool zooms in the image and crops out the portion outside the display area, and stretching the resulting cropped image to occupy the entire display area).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to further modify the combination of Kim et al. and Choe et al. with the teaching of Ubillos et al. such that the framing image is obtained by reducing or enlarging the edit image in accordance with the designated aspect ratio, so as to allow the user to select the aspect ratio of the output image while efficiently filling the available display area and preserving the selected image composition, thereby improving the functionality and usability of the device.
Claim 13 recites substantially similar subject matter as disclosed in claim 3; therefore, it is rejected for the same reasons.
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al. (US 2010/0173678 A1, cited in IDS) in view of Choe et al. (US 2021/0250510 A1) as applied to claim 2 above, and further in view of Parashar et al. (US 2021/0232921 A1).
As to claim 10, Kim et al. in view of Choe et al. discloses the electronic device of claim 1. The above combination fails to disclose the at least one processor includes a neural processing unit (NPU) and a central processing unit (CPU), and wherein the NPU is configured to include the framing engine.
However, Parashar et al. teaches the at least one processor includes a neural processing unit (NPU) and a central processing unit (CPU) (Fig.17), and wherein the NPU is configured to include the framing engine ([0209]: the NPU performs object detection on the input frames to detect the regions of interest).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to further modify the combination of Kim et al. and Choe et al. with the teaching of Parashar et al. such that the at least one processor includes a neural processing unit (NPU) and a central processing unit (CPU), and wherein the NPU is configured to include the framing engine, so as to accelerate neural network inference, improve camera frame process, and reduce the CPU processing burden.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZHENZHEN WU whose telephone number is (571)272-2519. The examiner can normally be reached 8:30 am - 5:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, SINH TRAN can be reached at (571)272-7564. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ZHENZHEN WU/Examiner, Art Unit 2637
/SINH TRAN/Supervisory Patent Examiner, Art Unit 2637
Prosecution Insights