DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
1. Acknowledgment is made of applicant's claim for foreign priority under 35 U.S.C. 119(a)-(d). Receipt is acknowledged of papers submitted under 35 U.S.C. 119(a)-(d), which papers have been placed of record in the file.
Information Disclosure Statement
2. The information disclosure statement (IDS) submitted on May 2, 2024, Jan 10, 2025, Oct 2,2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
3. Claim(s) 1,2,8,9,10 is/are rejected under 35 U.S.C. 102 (a)(2) as being anticipated by Lee et al. (US Pub. No.: US 2024/0143067 A1).
Regarding claim 1, Lee et al. discloses a wearable device (Para 58, 59; Fig. 3 A; the wearable device 300) comprising:
a camera (Para 65; the wearable device 300 may analyze an object included in a real image collected through a photographing camera 340-1, combine a virtual object corresponding to objects subject to augmented reality provision among the analyzed object and display it on the at least one display 350.) ;
an image signal processor (Para 75; The processor may perform a designated function based on identifying the signal corresponding to the operation. In an embodiment of the disclosure);
a display ( Para 85; A first display 350-1 for outputting an image to the left eye among both eyes of the user and a second display 350-2 for outputting an image to the right eye among the both eyes of the user may be disposed on the first surface 410.) ;
at least one processor ( Para 94; The processor 510 of the wearable device 101 according to an embodiment may include the hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), and/or an application processor (AP). ) comprising processing circuitry; and
memory ( Para 95; The memory 520 of the wearable device 101 may include the hardware component for storing data and/or instructions inputted and/or outputted to the processor 510.) , comprising one or more storage media, storing instructions,
wherein the instructions (Para 94; The processor 510 of the wearable device 101 according to an embodiment may include the hardware component for processing data based on one or more instructions.) that, when executed by the at least one processor individually or collectively, cause the electronic device to:
based on user input, identify a region of interest (ROI) from a first frame
displayed through the display ( Para 107-115; Fig. 6A-6C; icons 611,612,613,614 displayed in the list 610; the wearable device 101 may identify an input indicating mapping of the application in the list 610 to the external object included in the displaying area. The input may include a gesture overlapping the visual object (e.g., at least one of icons 611, 612, 613, and 614) representing a specific application included in the list 610. Referring to FIG. 6B, a state 602 in which the wearable device 101 identifies the input included in the list 610 is illustrated. In the state 602, the wearable device 101 may display the icon 612 overlapping the external object 622 in response to the input; the region wherein the icon 612 is moved to can be considered as region of interest), a camera region included in the first frame corresponding to at least partial region of a first camera image obtained through the camera at a first time point ( Para 107; Based on identifying external objects 621, 622, 623, and 624, the wearable device 101 may display bounding boxes 631, 632, 633, and 634 for guiding that the external objects 621, 622, 623, and 624 have been identified by the wearable device 101, in the displaying area; the camera region can be considered as the image of external object 621 surrounded by bounding box 631),
based on the ROI overlapping the camera region, control the image signal
processor to process a second camera image obtained at a second time point after
the first time point by a first processing scheme ( Fig. 6C; Para 117; when the icon 611 is moved to object 621 (region of icons and external object image areas are overlapped) ; the bounding box wrapping the object in a color different from that of the bounding boxes 633 and 634 in previous state and display the name “1App” of applications matched to the external object 621 ) ,
based on the ROI being outside of the camera region, control the image
signal processor to process the second camera image by a second processing
scheme that is distinct from the first processing scheme (Fig. 6B; Para 113-117l there is no icon on object 621 ( no overlapping of application/icon and external object 621 area); wherein only external object 622 is registered based on an input indicating that the icon 612 is dragged along the path 650. Based on the registration of the external object 622, the wearable device 101 may match a function (e.g., a function executed based on the attributes included in the window 660) to be executed by tracking the external object 622 to the external object 622. No object 621 is tracked; therefore, the processing scheme is different than Fig. 6C) , and
display, through the display, a second frame including the camera region
corresponding to at least partial region of the processed second camera image (Figs. 6D; Para 118-122; The wearable device 101 may display the attribute associated with at least one application selected through the list 692 in a portion 693 of the window 690. In the state 604 of FIG. 6D, the wearable device 101 may display the attribute associated with the drawing application in the portion 693, in response to an input indicating to select the drawing application for drawing an image in the list 692. For example, the wearable device 101 may display a visual object for adjusting a brush type corresponding to the external object 688 in the portion 693.) .
Regarding claim 2, Lee et al. discloses the wearable device of claim 1, comprising: a sensor (Para 72; The eye tracking camera 340-1 may output data representing the gaze of the user wearing the wearable device 300. For example, the wearable device 300 may detect the gaze from an image including the pupils of the user obtain through the eye tracking camera 340-1. An example of eye tracking camera 340-1 being disposed toward the right eye of the user is illustrated in FIG. 3B, but the embodiment is not limited thereto, and the eye tracking camera 340-1 may be disposed alone toward the left eye of the user or may be disposed toward both eyes.),
wherein the instructions that, when executed by the at least one processor
individually or collectively, cause the electronic device to:
obtain a gaze of a user identified through the sensor as the user input, and
identify the ROI based on the gaze of the user obtained as the user input (Para 111;
the wearable device 101 may identify the gesture based on a direction of the gaze of the user 110 toward the icon 612, and/or a period in which the direction of the gaze of the user 110 faces the icon 612. For example, the wearable device 101 may identify an input that maps a specific application to a specific external object based on the remarks (e.g., “Please map which application with which pencil”) of the user 110 and the gestures of the user 110 in the time section in which the remarks are received. Based on the input, the wearable device 101 may assign an application corresponding to the icon 612 to the external object 622 included in the displaying area).
Regarding claim 8, Lee et al. discloses The wearable device of claim 1,
wherein the instructions that, when executed by the at least one processor
individually or collectively, cause the electronic device to:
display, on the display, the second frame based on an application image
obtained by executing an application (Fig. 6D; Para 119; the wearable device 101 may display a visual object having the form of a window 682 including a designated text (e.g., “Please recognize the pen to be registered”) for guiding the selection of the external object in the displaying area. The wearable device 101 may display a bounding box 684 in association with the external object 688 included in the displaying area based on identifying the external object 688 in the displaying area.),
wherein the second frame includes the camera region based on the second camera
image and an application region based on the application image ( Para 118-119; a state 604 in which the wearable device 101 registers an external object 688 is illustrated by using a camera (e.g., a motion recognition camera 340-2 of FIG. 3B, cameras 440-9 and 440-10 of FIG. 4B and/or a camera 540 of FIG. 5) having FoV that at least partially overlaps the displaying area of the wearable device 101. The wearable device 101 may display lists of one or more external objects registered as the wearable device 101 in a visual object having the form of a window 670 overlapping at least a portion of the displaying area.), and
wherein the camera region is set based on a previously set camera region
indicated by usage information of the application ( Para 118-119; Fig. 6D; The wearable device 101 may display lists of one or more external objects registered as the wearable device 101 in a visual object having the form of a window 670 overlapping at least a portion of the displaying area. In the window 670, the wearable device 101 may display a visual object having the form of a button 672 including designated text (e.g., “add pen”) for registering the external object. In response to an input indicating to select the button 672, the wearable device 101 may display a UI for registering the external object in the displaying area. the wearable device 101 may display the UI for identifying the selection of the user for the external object in the displaying area based on the input indicating to select the button 672; The wearable device 101 may display a bounding box 684 in association with the external object 688 included in the displaying area based on identifying the external object 688 in the displaying area which can be done by previous input/usage information through the application) .
Regarding claim 9, Lee et al. discloses The wearable device of claim 1, wherein the instructions that, when executed by the at least one processor individually or collectively, cause the electronic device to:
identify at least one object included in the first camera image ( Fig. 6A; Para 107; he wearable device 101 may identify one or more external objects 621, 622, 623, and 624 shown by the user 110) ;
identify an object of a designated type from the identified at least one object ( Para 113; external object 622 recognized as a pen ) ; and
set the camera region based on a region of the object of the designated type ( Para 111-113; in case that the external object 622 is recognized as a pen including ink, the wearable device 101 may display the options in the window 660. The wearable device 101 may display options for selecting a portion of the external object 622 to be tracked by the wearable device 101 in the window 660F).
Regarding claim 10, the subject matter of claim 10 is similar to the subject matter disclosed in claim 1; therefore, claim 10 is rejected for the same reasons as set forth in claim 1.
Allowable subject matter
Claims 3-7 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 3, prior art on record Kimura (Pub. No.: US 2011/0157379 A1) discloses wherein the instructions that, when executed by the at least one processor individually or collectively, cause the electronic device to: in response to identifying that data indicating a change of pixel values of the camera region is out of a reference range (Para 159-160; The pixel shift distance d for a certain frame rate Fr is determined according to Expression 5 and Expression 7. Also, due to the nature of window processing, when a certain feature point has been shifted, motion detection cannot be performed unless the shifted position is in the range of the window. In view of this, giving dmax to a maximum shift distance for which motion detection can be performed, a limit rotation acceleration amax is acquired. For example, when Expression 5 is solved for a and d=dmax, the value of amax is defined in the following Expression 9. Similarly, when Expression 7 is solved for a, a value of the limit rotation acceleration value for rotation in the Roll direction can be obtained. This amax can be used as the threshold.) , control the image signal processor to obtain camera images for the camera region at a first frame per second (FPS) higher than a second FPS of the region ( Para 161; If the sensor output a' is greater than or equal to the threshold (S52: Yes), the frame rate change unit 32 changes the frame rate of the imaging unit 12 to a high frame rate Fr (a') (S53).),
and in response to identifying that the data is within the reference range,
control the image signal processor to obtain the camera images for the camera
region at the second FPS (Para 162; Note that if the sensor output a' is less than the threshold (S52: NO), the frame rate change unit 32 changes the frame rate of the imaging unit 12 back to the normal frame rate (S54).).
However, the prior art does not disclose “in response to identifying that data indicating a change of pixel values of the camera region which is overlapped with the ROI is out of a reference range, control the image signal processor to obtain the second camera image for the camera region at a first frame per second (FPS) higher than a second FPS of another region outside of the camera region of the second frame; and
in response to identifying that the data is within the reference range, control the
image signal processor to obtain the second camera image for the camera region at the
second FPS” in combination of other limitation in the claim.
Claims 4,5,6, are objected to as being dependent from claim 3.
Regarding claim 7, prior art on record Lee et al. (US Pub. No.: US 2024/0143067 A1) discloses the wearable device of claim 1, wherein the instructions that, when executed by the at least one processor individually or collectively, cause the electronic device to ( Para 94; The processor 510 of the wearable device 101 according to an embodiment may include the hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), and/or an application processor (AP). ):based on the ROI overlapping the camera region, control the image signal processor to process a second camera image obtained at a second time point after the first time point by a first processing scheme ( Fig. 6C; Para 117; when the icon 611 is moved to object 621 (region of icons and external object image areas are overlapped) ; the bounding box wrapping the object in a color different from that of the bounding boxes 633 and 634 in previous state and display the name “1App” of applications matched to the external object 621),
based on the ROI being outside of the camera region, control the image
signal processor to process the second camera image by a second processing
scheme that is distinct from the first processing scheme (Fig. 6B; Para 113-117l there is no icon on object 621 (no overlapping of application/icon and external object 621 area); wherein only external object 622 is registered based on an input indicating that the icon 612 is dragged along the path 650. Based on the registration of the external object 622, the wearable device 101 may match a function (e.g., a function executed based on the attributes included in the window 660) to be executed by tracking the external object 622 to the external object 622. No object 621 is tracked; therefore, the processing scheme is different than Fig. 6C) .
However, the prior art does not disclose “ in response to identifying that brightness of the camera region overlapped with the ROI is more than reference brightness, control the image signal processor to process the second camera image obtained at the second time point after the first time point by the first processing scheme; and
in response to identifying that brightness of the camera region overlapped with
the ROI is under reference brightness, control the image signal processor to process the second camera image for the camera region by the second processing scheme” in combination of other limitation in the claim and in its base claim.
Claims 11-20 are allowed.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 11, prior art on record Kimura (Pub. No.: US 2011/0157379 A1) discloses a wearable device (Para 59; Fig. 1; wearable camera) comprising:
a camera (Para 60; The wearable camera 10 includes an imaging unit 12);
an image signal processor ( Para 70; control unit 20) ;
a display (Fig. 3; windows that are displaying images);
at least one processor ( Para 188; process of an imaging device ) comprising processing circuitry; and
memory (Para 188-190; recording medium; The circulated or distributed control program may be provided for use by storage in a memory, etc. read by the processor, and the various functions described in the embodiments may be realized by executing the control program.), comprising one or more storage media, storing instructions,
wherein the instructions that, when executed by the at least one processor individually or collectively, cause the electronic device to:
display, on the display, a first frame, that includes, as a camera region, at
least partial region of a first camera image of environment surrounding the wearable device and obtained through the camera at a first time point (Para 84; frame image ) ,
during the first frame being displayed on the display, identify the camera
region within the first frame corresponding to a region of interest (Para 84; feature points),
in response to identifying that data indicating a change of pixel values of the camera region is out of a reference range (Para 159-160; The pixel shift distance d for a certain frame rate Fr is determined according to Expression 5 and Expression 7. Also, due to the nature of window processing, when a certain feature point has been shifted, motion detection cannot be performed unless the shifted position is in the range of the window. In view of this, giving dmax to a maximum shift distance for which motion detection can be performed, a limit rotation acceleration amax is acquired. For example, when Expression 5 is solved for a and d=dmax, the value of amax is defined in the following Expression 9. Similarly, when Expression 7 is solved for a, a value of the limit rotation acceleration value for rotation in the Roll direction can be obtained. This amax can be used as the threshold.), control the image signal processor to obtain camera images for the camera region at a first frame per second (FPS) higher than a second FPS of the region (Para 161; If the sensor output a' is greater than or equal to the threshold (S52: Yes), the frame rate change unit 32 changes the frame rate of the imaging unit 12 to a high frame rate Fr (a') (S53).), and in response to identifying that the data is within the reference range, control the image signal processor to obtain the camera images for the camera region at the second FPS (Para 162; Note that if the sensor output a' is less than the threshold (S52: NO), the frame rate change unit 32 changes the frame rate of the imaging unit 12 back to the normal frame rate (S54).).
However, the prior art does not disclose “in response to identifying that data indicating a change of pixel values of the camera region is out of a reference range, control the image signal processor to obtain camera images for the camera region at a first frame per second (FPS) higher than a second FPS of a region of the first frame outside of the camera region, and in response to identifying that the data is within the reference range, control the image signal processor to obtain the camera images for the camera region at the second FPS” in combination of other limitation in the claim.
Claims 12-19 are allowed as being dependent from claim 11.
Regarding claim 20, the subject matter of claim 20 is similar to the subject matter disclosed in claim 11; therefore, claim 20 is allowed for the same reasons as set forth in claim 11.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XI WANG whose telephone number is 469-295-9155. The examiner can normally be reached on Monday -Friday 8:00 AM-5:00 PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sinh Tran can be reached on 571-272-7564. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XI WANG/ Primary Examiner, Art Unit 2637
Prosecution Insights