Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Detailed Action
This is a Non-Final Office Action of the instant application 18/540,246 (hereinafter the ’246 application), which is a continuation of 16/894,593 (hereinafter the ’593 application), filed on 6/5/2020 and a reissue of US Patent Number 9,990,043 (hereinafter the ‘043 Patent) granted June 5, 2018. The ’593 application is a reissue of US Application No. 14/789,788 (hereinafter the ‘788 Application), filed July 1, 2015, which has been granted as the ‘043 Patent.
For reissue applications filed before September 16, 2012, all references to 35 U.S.C. 251 and 37 CFR 1.172, 1.175, and 3.73 are to the law and rules in effect on September 15, 2012. Where specifically designated, these are “pre-AIA ” provisions.
For reissue applications filed on or after September 16, 2012, all references to 35 U.S.C. 251 and 37 CFR 1.172, 1.175, and 3.73 are to the current provisions.
Obligations
Applicant is reminded of the continuing obligation under 37 CFR 1.178(b), to timely apprise the Office of any prior or concurrent proceed-ing in which Patent No. 9,990,043 is or was involved. These proceedings would include interferences, reissues, reexaminations, and litigation.
Applicant is further reminded of the continuing obligation under 37 CFR 1.56, to timely apprise the Office of any information which is mate-rial to patentability of the claims under consideration in this reissue appli-cation.
These obligations rest with each individual associated with the filing and prosecution of this application for reissue. See also MPEP §§ 1404, 1442.01 and 1442.04.
Applicant is notified that any subsequent amendment to the specification and/or claims must comply with 37 CFR 1.173(b).
Prosecution History
During initial examination, the claims were substantially amended from their original state to overcome the prior art of record [Arihara et al. (US PUB 20120146906), Forutanpour et al. (US PUB 20130335573), Prokoski (US PAT 7027621, Baldwin et al. (US PUB 20120256967)] Specifically, the Patent Owner added limitations corresponding to:
a thermographic camera having a first field of view, the thermographic camera to capture an infrared image within the first field of view;
an optical camera having a second field of view, the optical camera to detect an optical image, wherein the first field of view and the second field of view are substantially the same; and
a processor coupled to the thermographic camera, the processor to:
receive the infrared image from the thermographic camera;
receive the optical image from the optical camera;
determine whether a light condition of the optical image is a normal light condition, a low light condition, or a no light condition;
in response to the low light condition or the no light condition:
detect infrared radiation in the infrared image;
define a first readable zone within the infrared image, wherein the first readable zone is a portion of the first field of view that includes an image of a body part of a user;
identify a first gesture made by the user based on the infrared radiation within the first readable zone to obtain first gesture information; and
execute a first gesture recognition command associated with the first gesture information; and
in response to the normal light condition:
define a second readable zone within the optical image associated, wherein the second readable zone is a portion of the second field of view that includes the image of the body part of the user;
identify a second gesture made by the user within the second readable zone using optical recognition to obtain second gesture information;
execute a second gesture recognition command associated with the second gesture information.
See the claim set provided on 9/15/2017 and 2/21/2018 and the Examiner’s Amendment in the Notice of Allowance dated 3/28/2018 in the ‘788 application.
In the 3/28/2018 Reasons for allowance, the Examiner noted that “After the examiner thoroughly examined the prior art reference, updated searches, and with the newly reference found. Nowhere in the prior art references teach, suggest, or disclose…” that which is presented in the amendments (shown above). With respect to the new references, the Examiner noted:
Yu (U.S Pub: 20130155255) is cited to teach the camera control module is configured for controlling the optical camera to capture a photo, or controlling the infrared camera to record a video of a gesture of a user according to the current mode of the electronic device. The optical camera is used for automatically photography the real-time videos when the electronic device is in the composite mode, and for taking photos or photography videos according to the gestures of the user. The infrared camera is configured for capturing the real-time color temperature images of the gesture of the user when the electronic device is in the composite mode. When a human is in front of the infrared camera, the color temperature images of the human body captured by the infrared camera have a number of brightness levels and a number of color levels. The electronic device further comprises the image synthesis module is configured for synthesizing a composite image by using the real-time video photographed by the optical camera and the color temperature image of the gesture of the user captured by the infrared camera. The image synthesis module further calibrates the images according to the fixing positions and shooting angles of the optical camera and the infrared camera, and then synthesizes the composite image by using the calibrated images.
However, Yu does not explicitly disclose “...the optical camera determines a light condition based on normal light, low light, and no light. If the optical camera detected a low light or no light condition, then the infrared camera would detect the heat or radiation image of the user hand gesture in front of the optical/infrared camera in a low light or no light condition and can interpret/recognize the user hand gesture command within the detecting zone. If the optical camera detected a normal (ambient) light condition, then the optical camera would detect the image of the user hand gesture in front of the optical camera and can interpret/recognize the user hand gesture command within the detecting zone.
Chen et al (U.S Pub: 20120075463) is cited to teach a thermographic (infrared) camera and a visible light (optical) camera to detect heat or radiation temperature of the body based on the temperature even in the total darkness condition. The thermographic camera may identify an object’s depth and its surrounding using the object’s blackbody radiation profile then compare depth data obtained from the infrared sensor with the data obtained from its surrounding thermal imaging to determine the different spectrum of blackbody radiation and the background such as tracking movement of the objects.
However, Chen does not explicitly disclose “...the optical camera determines a light condition based on normal light, low light, and no light. If the optical camera detected a low light or no light condition, then the infrared camera would detect the heat or radiation image of the user hand gesture in front of the optical/infrared camera in a low light or no light condition and can interpret/recognize the user hand gesture command within the detecting zone. If the optical camera detected a normal (ambient) light condition, then the optical camera would detect the image of the user hand gesture in front of the optical camera and can interpret/recognize the user hand gesture command within the detecting zone.
Oath / Declaration
The reissue oath/declaration filed with this application is defective (see 37 CFR 1.175 and MPEP § 1414) because of the following:
Claims 1-38 are rejected under 35 U.S.C. 251 as being improperly broadened in a reissue application made and sworn to by the assignee. The application for reissue may be made and sworn to by the assignee of the entire interest only if the application does not seek to enlarge the scope of the claims of the original patent or, for reissue applications filed on or after September 16, 2012, the application for the original patent was filed by the assignee of the entire interest under 37 CFR 1.46.
A broadening reissue application must be applied for by all of the inventors (patentees), that is, the original reissue oath/declaration must be signed by all of the inventors. See MPEP § 1414.
[T]he inventor’s oath or declaration for a reissue application may be signed by the assignee of the entire interest if:
(1) The application does not seek to enlarge the scope of the claims of the original patent; or
(2) The application for the original patent was filed under § 1.46 by the assignee of the entire interest. See 37 C.F.R. 1.175 (c)
A claim is broader in scope than the original claims if it contains within its scope any conceivable product or process which would not have infringed the original patent. A claim is broadened if it is broader in any one respect even though it may be narrower in other respects.
Recapture
Claims 31-38 are rejected under 35 U.S.C. 251 as being an improper recapture of broadened claimed subject matter surrendered in the application for the patent upon which the present reissue is based and within related applications within the patent family. See Greenliant Systems, Inc. et al v. Xicor LLC, 692 F.3d 1261, 103 USPQ2d 1951 (Fed. Cir. 2012); In re Shahram Mostafazadeh and Joseph O. Smith, 643 F.3d 1353, 98 USPQ2d 1639 (Fed. Cir. 2011); North American Container, Inc. v. Plastipak Packaging, Inc., 415 F.3d 1335, 75 USPQ2d 1545 (Fed. Cir. 2005); Pannu v. Storz Instruments Inc., 258 F.3d 1366, 59 USPQ2d 1597 (Fed. Cir. 2001); Hester Industries, Inc. v. Stein, Inc., 142 F.3d 1472, 46 USPQ2d 1641 (Fed. Cir. 1998); In re Clement, 131 F.3d 1464, 45 USPQ2d 1161 (Fed. Cir. 1997); Ball Corp. v. United States, 729 F.2d 1429, 1436, 221 USPQ 289, 295 (Fed. Cir. 1984). A broadening aspect is present in the reissue which was not present in the application for patent. The record of the application for the patent shows that the broadening aspect (in the reissue) relates to claimed subject matter that applicant previously surrendered during the prosecution of the application. Accordingly, the narrow scope of the claims in the patent was not an error within the meaning of 35 U.S.C. 251, and the broader scope of claim subject matter surrendered in the application for the patent cannot be recaptured by the filing of the present reissue application.
During the prosecution of the ‘043 Patent (Application No. 14/789,788), the Patent owner argued that the following added limitations differentiated the claims from the prior art:
1 (Original). A system, comprising:
a thermographic camera having a first field of view, the thermographic camera to capture an infrared image within the first field of view;
an optical camera having a second field of view, the optical camera to detect an optical image, wherein the first field of view and the second field of view are substantially the same; and
a processor coupled to the thermographic camera, the processor to:
receive the infrared image from the thermographic camera; receive the optical image from the optical camera;
determine whether a light condition of the optical image is a normal light condition, a low light condition, or a no light condition;
in response to the low light condition or the no light condition:
detect infrared radiation in the infrared image;
define a first readable zone within the infrared image, wherein the first readable zone is a portion of the first field of view that includes an image of a body part of a user;
identify a first gesture made by the user based on the infrared radiation within the first readable zone to obtain first gesture information; and
execute a first gesture recognition command associated with the first gesture information; and
in response to the normal light condition:
define a second readable zone within the optical image associated, wherein the second readable zone is a portion of the second field of view that includes the image of the body part of the user;
identify a second gesture made by the user within the second readable zone using optical recognition to obtain second gesture information; and
execute a second gesture recognition command associated with the second gesture information.
See the claim set provided on 9/15/2017 and 2/21/2018 in the ‘788 application as well as the Examiner’s Amendment of 3/28/2018 following the Interview of 2/22/2018.
In considering this recapture rejection, the Examiner has evaluated the claims according to the recapture rule three-step process as set forth in In Clement, 131 F.3d at 1468-70, 45 USPQ2d at 1164-65:
PNG
media_image1.png
18
19
media_image1.png
Greyscale
(1) First, it was determined, in what respect, the reissue claims are broader in scope than the original patent claims. In the instant case the claims no longer include:
a thermographic camera having a first field of view, the thermographic camera to capture an infrared image within the first field of view;
an optical camera having a second field of view, the optical camera to detect an optical image, wherein the first field of view and the second field of view are substantially the same; and
Specifically claims 35-38 lack two separate cameras (namely thermographic and optical) being used to capture the two images.
a processor coupled to the thermographic camera, the processor to: receive the infrared image from the thermographic camera; receive the optical image from the optical camera;
Specifically claims 35-38 lack a processor for receiving and processing the images.
determine whether a light condition of the optical image is a normal light condition, a low light condition, or a no light condition;
Specifically claims 31-38 lack determining the light condition as one of the 3 specified light conditions. These 3 specific lighting conditions were added to the claims and are specific and when combined the claims do not limit in the same way. For example, if ‘normal’ is not specifically described but rather described as ‘neither a low-light condition nor a no-light condition’ then this excludes the case of a ‘high light’ condition where the image is oversaturated with light and not readable.
in response to the low light condition or the no light condition:
define a first readable zone within the infrared image, wherein the first readable zone is a portion of the first field of view that includes an image of a body part of a user;
identify a first gesture made by the user based on the infrared radiation within the first readable zone to obtain first gesture information; and
execute a first gesture recognition command associated with the first gesture information; and
Specifically claims 31-38 lack defining a first readable zone within the infrared image where a gesture is identified and command is executed based upon the gesture identification.
in response to the normal light condition:
define a second readable zone within the optical image associated, wherein the second readable zone is a portion of the second field of view that includes the image of the body part of the user;
identify a second gesture made by the user within the second readable zone using optical recognition to obtain second gesture information; and
execute a second gesture recognition command associated with the second gesture information.
Specifically claims 31-38 lack defining a second readable zone within the optical image where a gesture is identified and command is executed based upon the gesture identification.
(2) Next, we determine whether the broader aspects of the reissue claims relate to subject matter surrendered in the original prosecution. Here the below limitations were added to the claims on the above noted dates to distinguish the '788 Application over the prior art:
The claim set provided on 9/15/2017 and 2/21/2018 in the ‘788 application as well the Examiner’s Amendment of 3/28/2018 following the Interview of 2/22/2018 added the below underlined limitations:
a thermographic camera having a first field of view, the thermographic camera to capture an infrared image within the first field of view;
an optical camera having a second field of view, the optical camera to detect an optical image, wherein the first field of view and the second field of view are substantially the same; and
a processor coupled to the thermographic camera, the processor to:
receive the infrared image from the thermographic camera; receive the optical image from the optical camera;
determine whether a light condition of the optical image is a normal light condition, a low light condition, or a no light condition;
in response to the low light condition or the no light condition:
detect infrared radiation in the infrared image;
define a first readable zone within the infrared image, wherein the first readable zone is a portion of the first field of view that includes an image of a body part of a user;
identify a first gesture made by the user based on the infrared radiation within the first readable zone to obtain first gesture information; and
execute a first gesture recognition command associated with the first gesture information; and
in response to the normal light condition:
define a second readable zone within the optical image associated, wherein the second readable zone is a portion of the second field of view that includes the image of the body part of the user;
identify a second gesture made by the user within the second readable zone using optical recognition to obtain second gesture information; and
execute a second gesture recognition command associated with the second gesture information.
(3) Finally, we determine whether the reissue claims were materially narrowed in other respects, so that the claims may not have been enlarged, and hence avoid the recapture rule. In this regard, the claims were not materially narrowed in related aspects or were narrowed in unrelated aspects. Specifically:
No materially narrowing in other respects identified. The new claims further specify that the ‘system’ is a ‘smart phone’ and that the ‘body part’ is the user’s ‘face’, but these aspects had been previously prosecuted under dependent claims 4-5 and 9-10.
Different statutory categories do not in itself describe ‘overlooked aspects’, these new added claims appear to be directed to the same invention only being broadened.
These new added claims are clearly not unrelated to subject matter that was surrendered during the prosecution of the original application, and thus do not relate to overlooked aspects. (In re Youman)
As noted above, the Applicant asserted that the added and argued limitations patentably distinguish the claimed invention from the prior art. The reissue claims do not recite these added / argued limitations. Accordingly, claims 31-38 are rejected under 35 U.S.C. 251 for attempting to recapture surrendered subject matter.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4-6, 9-11, 13-14, 16-17, 19-30, and 31-38 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al., Patent Number: 9,202,105, hereinafter Wang in further view of Mangiat et al., Patent Number: 9,436,870, hereinafter Mangiat in further view of Bezos, Publication Number: 2010/0125816.
With regard to claims 1, 11, and 17 teaching a system, comprising: a thermographic camera having a first field of view, the thermographic camera to capture an infrared image within the first field of view; Wang teaches, in column 5, lines 1-24, in column 2, line 48 through column 3, line 5, and column 3, lines 38-45 and in figure 2, a thermographic camera having a field of view to capture an infrared image of the user in the field of view.
With regard to claims 1, 11, and 17 teaching an optical camera having a second field of view, the optical camera to detect an optical image, wherein the first field of view and the second field of view are substantially the same; Wang teaches, in column 2, line 48 through column 3, line 5, and column 3, lines 38-45 and in figure 1, an image capture camera having a field of view to capture an visual image of the user in the field of view.
With regard to claims 1, 11, and 17 teaching a processor coupled to the thermographic camera, the processor to: receive the infrared image from the thermographic camera; receive the optical image from the optical camera; Wang teaches, in column 13, lines 4-44 and in figure 8, a processor to receive captured infrared images and optical images for processing.
With regard to claims 1, 11, and 17 teaching determine whether a light condition of the optical image is a normal light condition, a low light condition, or a no light condition; Wang teaches, in column 10, line 61 through column 11, line 11 and in column 12, lines 20-42, determining the amount of light around the device to determine if there is sufficient lighting for an image to be analyzed.
With regard to claims 1, 11, and 17 teaching in response to the low light condition or the no light condition: detect infrared radiation in the infrared image; Wang teaches, in column 10, line 61 through column 11, line 11,upon determination that the default approach, optical image capture, cannot make a determination with at least a minimum level of confidence another approach, infrared capture, is utilized.
With regard to claims 1, 11, and 17 teaching define a first readable zone within the infrared image, wherein the first readable zone is a portion of the first field of view that includes an image of a body part of a user; Wang teaches, in column 2, line 48 through column 3, line 5, and column 3, lines 38-45, column 7, lines 50-67 and in figure 4, a defined zone in front of the camera which is a portion of the field of view that the interface can ask a user to position his or her face within for analysis.
With regard to claims 1, 11, and 17 teaching identify a first gesture made by the user based on the infrared radiation within the first readable zone to obtain first gesture information; Wang teaches, in column 5, lines 25-51 and column 7, lines 50-67 and in figure 4, identifying a gesture performed by a user captured within a defined region of the field of view.
With regard to claims 1, 11, and 17 teaching execute a first gesture recognition command associated with the first gesture information; Wang teaches, in column 5, lines 25-51 and column 13, lines 4-60 and in figures 4 and 8, identifying that a gesture performed by a user corresponds to a defined action to be executed upon identified correspondence.
With regard to claims 1, 11, and 17 teaching in response to the normal light condition (use optical capture), Wang teaches, in column 10, line 61 through column 11, line 11,upon determination that the default approach or optical image capture can make a determination with at least a minimum level of confidence use the default method, optical capture.
With regard to claims 1, 11, and 17 teaching define a second readable zone within the optical image associated, wherein the second readable zone is a portion of the second field of view that includes the image of the body part of the user; Wang teaches, in column 2, line 48 through column 3, line 5, and column 3, lines 38-45, column 7, lines 50-67 and in figure 4, a defined zone in front of the camera which is a portion of the field of view that the interface can ask a user to position his or her face within for analysis.
With regard to claims 1, 11, and 17 teaching identify a second gesture made by the user within the second readable zone using optical recognition to obtain second gesture information; Wang teaches, in column 5, lines 25-51 and column 7, lines 50-67 and in figure 4, identifying a gesture performed by a user captured within a defined region of the field of view.
With regard to claims 1, 11, and 17 teaching execute a second gesture recognition command associated with the second gesture information; Wang teaches, in column 5, lines 25-51 and column 13, lines 4-60 and in figures 4 and 8, identifying that a gesture performed by a user corresponds to a defined action to be executed upon identified correspondence.
Mangiat teaches a portable device for receiving visual input from a user using both optical imaging and infrared imaging (see column 4, line 57 through column 5, line 46), similar to that of Wang, but further teaches specifically transitioning to the IR based imaging when the optical imaging reveals a user’s face is insufficiently lite or in a dark environment (see also column 11, line 28 through column 13, line 11 and column 14, lines 17-47 and figures 1A, 1B, 6A, and 6B). It would have been obvious to one of ordinary skill in the art at the time of the invention to combine the optical image based interpretation of the image to determine if the lighting is insufficient or dark and to effect transition to use of IR illumination and detection accordingly, as did Mangiat, with face detection system taught by Wang. One would have been motivated to make said combination as both pieces of art are in the same field and stepwise escalation of imaging resources used makes the most sense from a power usage perspective.
Bezos teaches a similar facial recognition system to that of Wang and Mangiat, but further teaches recognizing a user’s facial gestures (tilt head, turn head, smile, wink, etc.) and executes specific corresponding commands to the recognized gestures (turn page, start navigation, confirm input, etc.) (see paragraphs 23-25, 29, 80, and 81 and figure 12a). Bezos further teaches tracking a user’s eyes monitoring, width, height, perceived relative separation, perceived relative position of the user’s eyes to the edge of a user’s face, etc. (see paragraph 65) and using the located features overtime to perform corresponding operations on the user interface (scroll, move cursor, etc.) (see paragraphs 67-69). It would have been obvious to one of ordinary skill in the art at the time of the invention to combine the facial recognition systems of Wang and Mangiat with the facial gesturing system executing commands specific to gestures of Bezos. One would have been motivated to make said combination as Wang and Mangiat already teach recognition of different gestures corresponding to different means of authentication, where combination with Bezos only further enforces the state of the art at the time as including commands to modify the interface based upon recognized facial gestures (see paragraph 25 of Bezos).
With regard to claims 4 and 13, teaching wherein the processor, the thermographic camera, and the optical camera are components of a mobile device, Wang teaches, in column 14, lines 3-20, the system for image analysis taking place in a mobile device.
With regard to claim 5, teaching wherein the mobile device is a smart phone, Wang teaches, in column 14, lines 3-20, the system for image analysis taking place in a cell phone / handheld messaging device / PDA.
With regard to claims 6 and 14, teaching wherein the processor, the thermographic camera, and the optical camera are components of a desktop computer or a laptop computer, Wang teaches, in column 14, lines 3-20, the system for image analysis taking place in a personal computer / laptop computer.
With regard to claims 9, 10, and 16, teaching wherein the first gesture or the second gesture is a head gesture, wherein the head gesture is at least one of a single-eye movement, a wink, a two-eye blink, an upward head movement, a downward head rightward head movement, or a leftward head movement, Wang teaches, in column 5, lines 25-50, the user gesturing to effect an operation of the device, wherein the gesture is of a head movement such as a wink, smile, frown, rotation of the user’s head, nodding, or performing another such action.
With regard to claims 19, 23, and 27 teaching wherein the first gesture and the second gestures are facial gestures, Wang teaches, in column 5, lines 25-50, the user using facial gestures to effect an operation of the device.
With regard to claims 20, 24, and 28 teaching wherein the facial gestures comprise facial motion, Wang teaches, in column 5, lines 25-50, the user gesturing to effect an operation of the device, wherein the gesture is of a head movement such as a wink, smile, frown, rotation of the user’s head, nodding, or performing another such action.
With regard to claims 21, 25, and 29 teaching wherein the facial gestures do not comprise facial motion Wang teaches, in column 7, line 41 through column 8, line 14, the user using their face orientation alone as input.
With regard to claims 22, 26, and 30 teaching wherein the processor, the thermographic camera, and the optical camera are components of a smart phone, Wang teaches, in column 14, lines 3-20, the system for image analysis taking place in a cell phone / handheld messaging device / PDA.
With regard to claims 31-38 teaching an image analysis via a smart phone method / device, Wang teaches, in column 14, lines 3-20, the system and method for image analysis taking place in a cell phone / handheld messaging device / PDA.
With regard to claims 31-38 teaching a thermographic camera having a first field of view, the thermographic camera configured to capture an infrared image within the first field of view; Wang teaches, in column 5, lines 1-24, in column 2, line 48 through column 3, line 5, and column 3, lines 38-45 and in figure 2, a thermographic camera having a field of view to capture an infrared image of the user in the field of view.
With regard to claims 31-38 teaching an optical camera having a second field of view, the optical camera configured to capture an optical image within the second field of view, wherein the first field of view and the second field of view are substantially the same; Wang teaches, in column 2, line 48 through column 3, line 5, and column 3, lines 38-45 and in figure 1, an image capture camera having a field of view to capture an visual image of the user in the field of view.
With regard to claims 31-38 teaching determine whether a light condition of the optical image is a normal light condition, a low light condition, or a no light condition; Wang teaches, in column 10, line 61 through column 11, line 11 and in column 12, lines 20-42, determining the amount of light around the device to determine if there is sufficient lighting for an image to be analyzed. Wang further teaches, in column 10, line 61 through column 11, line 11,upon determination that the default approach, optical image capture, cannot make a determination with at least a minimum level of confidence another approach, infrared capture, is utilized.
With regard to claims 31-38 teaching a processor coupled to the thermographic camera and the optical camera, the processor configured to: determine if a light condition is a low-light condition or a no-light condition (not normal); Wang teaches, in column 13, lines 4-44 and in figure 8, a processor to receive captured infrared images and optical images for processing. Where Wang further teaches, in column 10, line 61 through column 11, line 11 and in column 12, lines 20-42, determining the amount of light around the device to determine if there is sufficient lighting for an image to be analyzed.
With regard to claims 31-38 teaching to detect infrared radiation in the infrared image, Wang further teaches, in column 4, lines 8-29, us or an infrared detector to detect light emitted by the infrared LEDs as reflected off of the subject.
With regard to claims 31-38 teaching in response to a determination that the light condition is neither a low-light condition nor a no-light condition (normal), process data representing a second readable zone of the optical image to identify a facial gesture of the user of the smart phone, wherein the second readable zone includes an image of the face of the user, Wang teaches, in column 10, line 61 through column 11, line 11, upon determination that the default approach, optical image capture, cannot make a determination with at least a minimum level of confidence because of poor lighting another approach, infrared capture, is utilized. Wang further teaches, in column 2, line 48 through column 3, line 5, and column 3, lines 38-45, column 7, lines 50-67 and in figure 4, a defined zone in front of the camera which is a portion of the field of view that the interface can ask a user to position his or her face within for analysis. Wang further teaches, in column 5, lines 25-51 and column 7, lines 50-67 and in figure 4, identifying a facial gesture performed by a user captured within a defined region of the field of view.
With regard to claims 31-38 teaching in response to a determination that the light condition is either a low-light condition or a no-light condition, process data representing a first readable zone of the infrared image to identify a facial gesture of a user of the smart phone, wherein the first readable zone includes an image of the face of the user, Wang teaches, in column 10, line 61 through column 11, line 11, upon determination that the default approach or optical image capture can make a determination with at least a minimum level of confidence based upon sufficient lighting the system utilizes the default method, optical capture. Wang further teaches, in column 2, line 48 through column 3, line 5, and column 3, lines 38-45, column 7, lines 50-67 and in figure 4, a defined zone in front of the camera which is a portion of the field of view that the interface can ask a user to position his or her face within for analysis. Wang further teaches, in column 5, lines 25-51 and column 7, lines 50-67 and in figure 4, identifying a gesture performed by a user captured within a defined region of the field of view.
With regard to claims 31-38 teaching take an action based on the identified facial gesture of the user of the smart phone, Wang teaches, in column 5, lines 25-51 and column 13, lines 4-60 and in figures 4 and 8, identifying that a gesture performed by a user corresponds to a defined action to be executed upon identified correspondence.
Mangiat teaches a portable device for receiving visual input from a user using both optical imaging and infrared imaging (see column 4, line 57 through column 5, line 46), similar to that of Wang, but further teaches specifically transitioning to the IR based imaging when the optical imaging reveals a user’s face is insufficiently lite or in a dark environment (see also column 11, line 28 through column 13, line 11 and column 14, lines 17-47 and figures 1A, 1B, 6A, and 6B). It would have been obvious to one of ordinary skill in the art at the time of the invention to combine the optical image based interpretation of the image to determine if the lighting is insufficient or dark and to effect transition to use of IR illumination and detection accordingly, as did Mangiat, with face detection system taught by Wang. One would have been motivated to make said combination as both pieces of art are in the same field and stepwise escalation of imaging resources used makes the most sense from a power usage perspective.
Bezos teaches a similar facial recognition system to that of Wang and Mangiat, but further teaches recognizing a user’s facial gestures (tilt head, turn head, smile, wink, etc.) and executes specific corresponding commands to the recognized gestures (turn page, start navigation, confirm input, etc.) (see paragraphs 23-25, 29, 80, and 81 and figure 12a). Bezos further teaches tracking a user’s eyes monitoring, width, height, perceived relative separation, perceived relative position of the user’s eyes to the edge of a user’s face, etc. (see paragraph 65) and using the located features overtime to perform corresponding operations on the user interface (scroll, move cursor, etc.) (see paragraphs 67-69). It would have been obvious to one of ordinary skill in the art at the time of the invention to combine the facial recognition systems of Wang and Mangiat with the facial gesturing system executing commands specific to gestures of Bezos. One would have been motivated to make said combination as Wang and Mangiat already teach recognition of different gestures corresponding to different means of authentication, where combination with Bezos only further enforces the state of the art at the time as including commands to modify the interface based upon recognized facial gestures (see paragraph 25 of Bezos).
Claims 2-3, 7-8, 12, 15, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al., Patent Number: 9,202,105, hereinafter Wang, Mangiat et al., Patent Number: 9,436,870, hereinafter Mangiat, and Bezos, Publication Number: 2010/0125816, as in claim 1 above, and in further view of Forutanpour et al., Publication Number: 2013/0335573, hereinafter Forutanpour.
With regard to claims 2, 3, 12, and 18, teaching wherein the processor, the thermographic camera, and the optical camera are supported on an eyewear article frame and use in a augmented reality system, Wang, Mangiat, and Bezos teach the above referenced system for implementing body detection using either optical or infrared based capture (supra), where Mangiat does note use of such personal electronic devices giving examples of glasses and head-mounted displays, but does not specifically teach use of the system in eyewear of the user. Forutanpour teaches a system using cameras and infrared sensors to locate a user’s body in space and effect actions dependent on a user’s movements (see paragraphs 53 and 54), similar to that of Wang, Mangiat, and Bezos, but further teaches use of the system within eyewear worn by the user (see paragraphs 35, 36, 39, 48, and 49 and figure 7). Here a user’s eyewear captures movements of a user’s hands in front of the glasses and performs actions on the augmented reality environment based upon a preprogramed correspondence between the movements and actions. It would have been obvious to one of ordinary skill in the art at the time of the invention to use the image based input system of Wang, Mangiat, and Bezos in a glasses based system as did Forutanpour. One would have been motivated to make said combination as each piece of art is in the same field and the combination allows for input within an even more immersive form of portable electronic device.
With regard to claims 7, 8, and 15 teaching wherein the first gesture or the second gesture is a hand gesture, wherein the hand gesture is at least one of an open hand gesture, a closed hand gesture, a swipe right, a swipe left, a swipe forward, or a swipe backward, Wang, Mangiat, and Bezos teach the above referenced system for implementing body detection using either optical or infrared based capture (supra), but does not specifically teach use of the system for reading hand motions. Forutanpour teaches a system using cameras and infrared sensors to locate a user’s body in space and effect actions dependent on a user’s movements (see paragraphs 53 and 54), similar to that of Wang, Mangiat, and Bezos, but further teaches use of the system for interpreting hand motions of a user including swipes in various directions and how a hand is open or closed (see paragraphs 60-62, 87, 89, and 102 and in figure 7). Here a user’s eyewear captures movements of a user’s hands in front of the glasses and performs actions on the augmented reality environment based upon a preprogramed correspondence between the movements and actions. It would have been obvious to one of ordinary skill in the art at the time of the invention to use the image based input system of Wang, Mangiat, and Bezos in a hand based input system as did Forutanpour. One would have been motivated to make said combination as each piece of art is in the same field and the combination allows for input within an even more immersive form of portable electronic device, with the hands enabling a greater variety of input.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DENNIS G BONSHOCK whose telephone number is (571)272-4047. The examiner can normally be reached on M-F 7:15 - 4:45.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexander Kosowski can be reached on 571-272-3744. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DENNIS G BONSHOCK/Primary Examiner, Art Unit 3992
Conferees:
/William H. Wood/ Reexamination Specialist, Art Unit 3992
/ALEXANDER J KOSOWSKI/Supervisory Patent Examiner, Art Unit 3992
Prosecution Insights