DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lopez et al. (US Patent Application Publication No. 2016/0274762 A1) in view of Hilliges et al. (US Patent Application Publication No. 2012/0113223).
Regarding claim 1, Lopez discloses an appliance computing device comprising a processor operably coupled to an imaging device, the processor configured to execute instructions that cause operations, the operations comprising [see para. 0021; a user of a computing device (e.g., an augmented reality or virtual reality device) interact with and control objects and applications displayed on the computing device through the user's eye movement. An image of the user's eyes or face, captured by one or more face-facing cameras on or coupled to the computing device which corresponds to camera generate virtual UI];
generating a virtual user interface relative to a surface within a field of view of the imaging device [see para. 0026, 0048 and figure 10; the camera module is used to identify the objects present in the field of view of the user and the AR device is made aware of the target devices present in the environment, in the user's field of view. The AR device may use forward-pointing cameras to locate the one or more physical devices present in the environment. This can be done by using image recognition techniques, by employing infrared markers; which corresponds to camera generate virtual UI on viewed appliance], the virtual user interface comprising a digital marker corresponding to an appliance control function [see para, 0057-0059 and figures 8 and 11; the UI is presented for a refrigerator. The UI indicates the current temperature of the refrigerator and also provides information regarding the contents of the refrigerator and providing interaction with target devices using an AR device; which corresponds to generate the virtual UI with digital marker for appliance control function]; however, Lopez fails to explicitly teach generating a control signal based on a physical user interaction relative to the digital marker.
Hilliges discloses generating a control signal based on a physical user interaction relative to the digital marker [see para. 0055-0058 and figures 5-8, an augmented reality environment in which the user is performing a gesture for virtual object creation. The augmented reality environment comprises a virtual object in the form a surface on which the user use a digit of hand to trace an arbitrary shape. The traced shape serves as "blue print" for an extrusion interaction. If the user makes a pinch gesture by bringing together the thumb and forefinger, then this gesture detected by the computing device and a gesture-based interaction technique can operate in a similar scenario can recognize the gesture of a given finger coming into contact with (e.g. tapping) the virtual object located on the user's palm, and consequently trigger the function associated with the given finger. This can apply the associated function to the virtual object executing a copy operation on the virtual object if the thumb is tapped on the virtual object; which corresponds to generate of the physical gesture representation to the virtual marker].
It would have been obvious to one of an ordinary skill in the art, having the teachings of Lopez and Hilliges before the affective filing date of the claimed invention to modify, augmented reality device interaction control system of Lopez to combine user interaction in augmented reality, as taught by Hilliges. One would have been motivated to make such a combination in order improve accuracy of gesture detection in augmented reality device system for better user interaction to appliance control function.
Regarding claims 2 and 10, Lopez discloses the operations comprising: obtaining an image comprising a digital representation of the physical user interaction within the field of view of the imaging device [see para. 0048; the AR device is made aware of the target devices present in the environment, in the user's field of view. The AR device may use forward-pointing cameras to locate the one or more physical devices present in the environment. This can be done by using image recognition techniques, by employing infrared markers].
Hilliges discloses determining a spatial correlation of the digital marker to the digital representation [see para. 0031; the object representation is spatially aligned with the view of the real object that the user can see on the display device, and the object representation may or may not be visible to the user on the display device. The object representation can, in one example, be a computer-derived virtual representation of a body part or other object, is a mesh or point-cloud object directly derived from the camera images. As the user moves the real object, the object representation moves in a corresponding manner in the augmented reality environment. One would have been motivated to make such a combination in order improve accuracy of gesture detection in augmented reality device system for better user interaction to appliance control function.
Regarding claims 3 and 11, Lopez discloses wherein determining the spatial correlation of the digital marker to the digital representation comprises: segmenting the image into one or more parts corresponding to the digital marker [see para. 0060; the UI is controlled using hand gestures. For example, a particular hand gesture associated with a particular menu option. As another example, the user may be able to use a finger to “touch” a presented menu option by placing the finger so that it occupies the same real location as the menu option appears to occupy, as presented by the AR device. In response to detecting the “touch,” the menu option may be activated]; and determining, at the one or more parts, an overlay of the digital representation of the physical user interaction to the digital marker [see para. 0049; The UI may be presented to the user as an overlay next to or on top of the physical device when the user's gaze is directed at the device, the UI is presented when the AR device is pointed toward the physical device, without regard to the point of regard of the user; which corresponds to the digital marker overlay of the digital representation of the physical user interaction].
Regarding claims 4 and 12, Hilligies discloses the operations comprising: determining the physical user interaction relative to the digital marker based on the overlay of the digital representation of the physical user interaction to the digital marker [see para. 0066; The user's own hands are visible through the optical beam splitter, and by visually aligning the augmented reality environment 102 and the user's hand (using camera) it can appear to the user that their real hands are directly manipulating the virtual objects. Virtual objects and controls rendered so that they appear superimposed on the user's hands and move with the hands, enabling the haptic feedback technique, and the camera enables the pose of the hands to be tracked and gestures recognized; which corresponds to represent of the physical interaction to marker based on overlay].
Regarding claims 5 and 13, Lopez discloses the operations comprising: superimposing the digital marker relative to the surface [see para. 0051 and figures 4-5; a UI for the thermostat superimposed on top of the thermostat. The UI in response to a user command in the form of the user's gaze being directed to the thermostat, the AR device being pointed at the thermostat (e.g., through a head rotation of the user or using a controller)].
Regarding claim 6, Lopez discloses wherein the control signal corresponds to the appliance control function [see para. 0070 and figure 6; a UI displayed in response to a signal received over a network; this additional UI hides the UI for the thermostat. In this example, the signal is an incoming video call and the UI indicates the type of the signal and information about the signal. The user may respond to the signal by looking at a device and issuing a voice command. The app for the TV may respond to the command and the context to send a request to the video-call system for video data to be sent to the TV. Accordingly, video for the call will be displayed on the TV instead of within the AR device].
Regarding claim 7, Lopez discloses the operations comprising: transmitting the control signal to articulate an operational component corresponding to the appliance control function [see para. 0033; an eye tracking engine when an application starts or stops the eye tracking engine; an external slider; dedicated on-off button on the AR device; an application or a digital button on the display; movement or shaking of the AR device; voice commands; on-screen capacitive buttons; touch pads; remote wireless control; or any suitable combination thereof. The camera modules are controlled by an application using the eye tracking features. As such, the eye tracking components consume power only while the camera is turned on (e.g., when the user is using the eye tracking features)].
Regarding claims 8 and 14, Lopez discloses the operations comprising: projecting, toward the surface, the virtual user interface as a light-based representation perceptible to a user [see para. 0027, 0034; The AR device include one or more light sources such as light-emitting diodes (LEDs). The one or more light sources are infrared LEDs. The one or more light sources are infrared lasers. The one or more infrared lasers emit structured light that allows the one or more camera modules to reconstruct a 3D scene].
Regarding claim 9, Lopez discloses a method for operating an appliance, the method comprising: [see para. 0021; a user of a computing device (e.g., an augmented reality or virtual reality device) interact with and control objects and applications displayed on the computing device through the user's eye movement. An image of the user's eyes or face, captured by one or more face-facing cameras on or coupled to the computing device which corresponds to camera generate virtual UI];
generating a virtual user interface relative to a surface within a field of view of the imaging device [see para. 0026, 0048 and figure 10; the camera module is used to identify the objects present in the field of view of the user and the AR device is made aware of the target devices present in the environment, in the user's field of view. The AR device may use forward-pointing cameras to locate the one or more physical devices present in the environment. This can be done by using image recognition techniques, by employing infrared markers; which corresponds to camera generate virtual UI on viewed appliance], the virtual user interface comprising a digital marker corresponding to an appliance control function [see para, 0057-0059 and figures 8 and 11; the UI is presented for a refrigerator. The UI indicates the current temperature of the refrigerator and also provides information regarding the contents of the refrigerator and providing interaction with target devices using an AR device; which corresponds to generate the virtual UI with digital marker for appliance control function]; and transmitting the control signal to articulate an operational component corresponding to the appliance control function [see para. 0033; an eye tracking engine when an application starts or stops the eye tracking engine; an external slider; dedicated on-off button on the AR device; an application or a digital button on the display; movement or shaking of the AR device; voice commands; on-screen capacitive buttons; touch pads; remote wireless control; or any suitable combination thereof. The camera modules are controlled by an application using the eye tracking features. As such, the eye tracking components consume power only while the camera is turned on (e.g., when the user is using the eye tracking features)]; however, Lopez fails to explicitly teach generating a control signal based on a physical user interaction relative to the digital marker.
Hilliges discloses generating a control signal based on a physical user interaction relative to the digital marker [see para. 0055-0058 and figures 5-8, an augmented reality environment in which the user is performing a gesture for virtual object creation. The augmented reality environment comprises a virtual object in the form a surface on which the user a digit of hand to trace an arbitrary shape. The traced shape serves as "blue print" for an extrusion interaction. If the user makes a pinch gesture by bringing together the thumb and forefinger, then this gesture detected by the computing device and a gesture-based interaction technique can operate in a similar scenario can recognize the gesture of a given finger coming into contact with (e.g. tapping) the virtual object located on the user's palm, and consequently trigger the function associated with the given finger. This can apply the associated function to the virtual object executing a copy operation on the virtual object if the thumb is tapped on the virtual object; which corresponds to generate of the physical gesture representation to the virtual marker]
It would have been obvious to one of an ordinary skill in the art, having the teachings of Lopez and Hilliges before the affective filing date of the claimed invention to modify, augmented reality device interaction control system of Lopez to combine user interaction in augmented reality, as taught by Hilliges. One would have been motivated to make such a combination in order improve accuracy of gesture detection in augmented reality device system for better user interaction to appliance control function.
Regarding claim 15, Lopez discloses an appliance [see para. 0046 and figure 4; kitchen appliances], comprising: a surface [see para. 0057; the UI indicates the current temperature of the refrigerator and also provides information regarding the contents of the refrigerator];
an operational component configured to perform a control function [see para. 0033; the camera modules are controlled by an application using the eye tracking features. As such, in some embodiments, the eye tracking components consume power only while the camera is turned on (e.g., when the user is using the eye tracking features)];
an imaging device configured with a field of view toward the surface [see para. 0026, 0048 and figure 10; the camera module is used to identify the objects present in the field of view of the user and the AR device is made aware of the target devices present in the environment, in the user's field of view. The AR device may use forward-pointing cameras to locate the one or more physical devices present in the environment. This can be done by using image recognition techniques, by employing infrared markers; which corresponds to camera generate virtual UI on viewed appliance];
a computing device comprising a processor operably coupled to the imaging device, the processor configured to execute operations, the operations comprising [see para. 0041 and figure 2; The camera control module also include the circuitry necessary to process the images delivered by the camera module. The camera control module include a processor that may optimize image quality, detect regions of interest on the image delivered by the camera module the image delivered by the camera module];
generating a virtual user interface relative to a surface within a field of view of the imaging device [see para. 0026, 0048 and figure 10; the camera module is used to identify the objects present in the field of view of the user and the AR device is made aware of the target devices present in the environment, in the user's field of view. The AR device may use forward-pointing cameras to locate the one or more physical devices present in the environment. This can be done by using image recognition techniques, by employing infrared markers; which corresponds to camera generate virtual UI on viewed appliance], the virtual user interface comprising a digital marker corresponding to an appliance control function [see para, 0057-0059 and figures 8 and 11; the UI is presented for a refrigerator. The UI indicates the current temperature of the refrigerator and also provides information regarding the contents of the refrigerator and providing interaction with target devices using an AR device; which corresponds to generate the virtual UI with digital marker for appliance control function]; and transmitting the control signal to articulate an operational component corresponding to the appliance control function [see para. 0033; an eye tracking engine when an application starts or stops the eye tracking engine; an external slider; dedicated on-off button on the AR device; an application or a digital button on the display; movement or shaking of the AR device; voice commands; on-screen capacitive buttons; touch pads; remote wireless control; or any suitable combination thereof. The camera modules are controlled by an application using the eye tracking features. As such, the eye tracking components consume power only while the camera is turned on (e.g., when the user is using the eye tracking features)]; however, Lopez fails to explicitly teach generating a control signal based on a physical user interaction relative to the digital marker.
Hilliges discloses generating a control signal based on a physical user interaction relative to the digital marker [see para. 0055-0058 and figures 5-8, an augmented reality environment in which the user is performing a gesture for virtual object creation. The augmented reality environment comprises a virtual object in the form a surface on which the user a digit of hand to trace an arbitrary shape. The traced shape serves as "blue print" for an extrusion interaction. If the user makes a pinch gesture by bringing together the thumb and forefinger, then this gesture detected by the computing device and a gesture-based interaction technique can operate in a similar scenario can recognize the gesture of a given finger coming into contact with (e.g. tapping) the virtual object located on the user's palm, and consequently trigger the function associated with the given finger. This can apply the associated function to the virtual object executing a copy operation on the virtual object if the thumb is tapped on the virtual object; which corresponds to generate of the physical gesture representation to the virtual marker]
It would have been obvious to one of an ordinary skill in the art, having the teachings of Lopez and Hilliges before the affective filing date of the claimed invention to modify, augmented reality device interaction control system of Lopez to combine user interaction in augmented reality, as taught by Hilliges. One would have been motivated to make such a combination in order improve accuracy of gesture detection in augmented reality device system for better user interaction to appliance control function.
Regarding claim 16, Hilliges discloses wherein the surface comprises a physical marker, and wherein the physical marker is uncoupled electrically to the computing device, and wherein the operations comprise superimposing the digital marker to the physical marker at the surface [see para. 0038; the computing device use the corresponding position and orientation to render a virtual object that maintains a relative spatial relationship with the first object. For example, the virtual object can be rendered superimposed on (i.e. coincident with) or around the first object, and the virtual object moves (and optionally rotates, scales and translates) with the movement of the first object].
Regarding claim 17, Lopez discloses the operations comprising: obtaining an image comprising a digital representation of the physical user interaction within the field of view of the imaging device [see para. 0048; the AR device is made aware of the target devices present in the environment, in the user's field of view. The AR device may use forward-pointing cameras to locate the one or more physical devices present in the environment. This can be done by using image recognition techniques, by employing infrared markers].
Hilliges discloses determining a spatial correlation of the digital marker to the digital representation [see para. 0031; the object representation is spatially aligned with the view of the real object that the user can see on the display device, and the object representation may or may not be visible to the user on the display device. The object representation can, in one example, be a computer-derived virtual representation of a body part or other object, is a mesh or point-cloud object directly derived from the camera images. As the user moves the real object, the object representation moves in a corresponding manner in the augmented reality environment. One would have been motivated to make such a combination in order improve accuracy of gesture detection in augmented reality device system for better user interaction to appliance control function.
Regarding claim 18, Lopez discloses wherein determining the spatial correlation of the digital marker to the digital representation comprises: segmenting the image into one or more parts corresponding to the digital marker [see para. 0060; the UI is controlled using hand gestures. For example, a particular hand gesture associated with a particular menu option. As another example, the user may be able to use a finger to “touch” a presented menu option by placing the finger so that it occupies the same real location as the menu option appears to occupy, as presented by the AR device. In response to detecting the “touch,” the menu option may be activated]; and determining, at the one or more parts, an overlay of the digital representation of the physical user interaction to the digital marker [see para. 0049; The UI may be presented to the user as an overlay next to or on top of the physical device when the user's gaze is directed at the device, the UI is presented when the AR device is pointed toward the physical device, without regard to the point of regard of the user; which corresponds to the digital marker overlay of the digital representation of the physical user interaction].
Regarding claim 19, Hilligies discloses the operations comprising: determining the physical user interaction relative to the digital marker based on the overlay of the digital representation of the physical user interaction to the digital marker [see para. 0066; The user's own hands are visible through the optical beam splitter, and by visually aligning the augmented reality environment 102 and the user's hand (using camera) it can appear to the user that their real hands are directly manipulating the virtual objects. Virtual objects and controls rendered so that they appear superimposed on the user's hands and move with the hands, enabling the haptic feedback technique, and the camera enables the pose of the hands to be tracked and gestures recognized; which corresponds to represent of the physical interaction to marker based on overlay].
Regarding claim 20, Lopez discloses wherein determining the spatial correlation of the digital marker to the digital representation comprises: segmenting the image into one or more parts corresponding to the digital marker [see para. 0060; the UI is controlled using hand gestures. For example, a particular hand gesture associated with a particular menu option. As another example, the user may be able to use a finger to “touch” a presented menu option by placing the finger so that it occupies the same real location as the menu option appears to occupy, as presented by the AR device. In response to detecting the “touch,” the menu option may be activated]; and determining, at the one or more parts, an overlay of the digital representation of the physical user interaction to the digital marker [see para. 0049; The UI may be presented to the user as an overlay next to or on top of the physical device when the user's gaze is directed at the device, the UI is presented when the AR device is pointed toward the physical device, without regard to the point of regard of the user; which corresponds to the digital marker overlay of the digital representation of the physical user interaction]; and a physical marker at the surface [see para. 0046 and figure 4].
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure (See PTO-892).
Lee et al. (US2024/0419304) discloses a method of controlling a household appliance includes setting, within an augmented reality note, control commands for the household appliance. The method also included receiving, on a remote user interface device, the augmented reality note. The method further includes activating the household appliance based on the control commands set within the received augmented reality note.
Drake (US 10,504,384) discloses the user engagement system may include a camera assembly, an image monitor, and a controller. The camera assembly may be directed at a cooking zone. The image monitor may be directed at an area forward from the cooking zone. The controller may be in operative communication with the camera assembly and the image monitor. The controller may be configured to initiate a directed cooking operation. The directed cooking operation may include receiving a video signal of the cooking zone from the camera assembly, presenting a real-time feed of the cooking zone at the image monitor, overlaying a dynamic figure onto the real-time feed of the cooking zone at the image monitor, and animating the dynamic figure over the real-time feed based on a selected recipe.
A reference to specific paragraphs, columns, pages, or figures in a cited prior art reference is not limited to preferred embodiments or any specific examples. It is well settled that a prior art reference, in its entirety, must be considered for all that it expressly teaches and fairly suggests to one having ordinary skill in the art. Stated differently, a prior art disclosure reading on a limitation of Applicant's claim cannot be ignored on the ground that other embodiments disclosed were instead cited. Therefore, the Examiner's citation to a specific portion of a single prior art reference is not intended to exclusively dictate, but rather, to demonstrate an exemplary disclosure commensurate with the specific limitations being addressed. In re Heck, 699 F.2d 1331, 1332-33,216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006,1009, 158 USPQ 275, 277 (CCPA 1968)). In re: Upsher-Smith Labs. v. Pamlab, LLC, 412 F.3d 1319, 1323, 75 USPQ2d 1213, 1215 (Fed. Cir. 2005); In re Fritch, 972 F.2d 1260, 1264, 23 USPQ2d 1780, 1782 (Fed. Cir. 1992); Merck & Co. v. Biocraft Labs., Inc., 874 F.2d 804, 807, 10 USPQ2d 1843, 1846 (Fed. Cir. 1989); In re Fracalossi, 681 F.2d 792,794 n.1,215 USPQ 569, 570 n.1 (CCPA 1982); In re Lamberti, 545 F.2d 747, 750, 192 USPQ 278, 280 (CCPA 1976); In re Bozek, 416 F.2d 1385, 1390, 163 USPQ 545, 549 (CCPA 1969).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CAO H NGUYEN whose telephone number is (571)272-4053. The examiner can normally be reached on Mon-Fri 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kieu Vu can be reached on 571-272-4057. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CAO H NGUYEN/ Primary Examiner, Art Unit 2171
Prosecution Insights