Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6, 21-31, and 33 are rejected under 35 U.S.C. 103 as being unpatentable over Nixon (K. W. Nixon, X. Chen and Yiran Chen, "Scope - quality retaining display rendering workload scaling based on user-smartphone distance," 2016 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), Austin, TX, USA, 2016, pp. 1-6, doi: 10.1145/2966986.2967073) in view of Krulce (Pub No. US 20140267034 A1).
As per claim 1, Nixon teaches the claimed:
1. An apparatus comprising: processing circuitry coupled to a memory, the processing circuitry to :receive an input from one or more detectors proximate a display to present an output from a graphics pipeline; (Nixon teaches this being used for a graphics pipeline for rendering. Nixon Abstract: “Modern smartphone display system come equipped with powerful GPU's capable of rendering advanced 2D and 3D graphics. These GPU's make up a significant portion of the system power profile due to the high resolution and framerate of smartphone display. These display features are selected during the design phase of a smartphone and correspond to the capabilities of the human visual system (HVS). However, the level of detail observable by the HVS is not static and changes with user-smartphone distance. In this paper we propose Scope, a system which alters the rendering resolution and framerate on a smartphone to scale display rendering workload in response to real time changes in user-smartphone distance. We demonstrate a new method of measuring this distance in real-time which is able to minimize front-facing camera sampling through the use of sensor fusion techniques. The result is that Scope requires a power overhead of only 20mW on average as the front-facing camera need only be sampled 4 times per minute. We demonstrate that Scope is able to reduce smartphone power consumption up to 58% while retaining visual quality in most cases.” The sensors are what determine the user-smartphone distance. The level of detail pertains to the output to the graphics pipeline.).
Nixon alone does not explicitly teach the remaining claim limitations.
However, Nixon in combination with Krulce teaches the claimed:
in response to determining that a user is not interacting with the display, reduce a frame rendering rate of the graphics pipeline. (Krulce [0008]: “Consistent with some embodiments, there is further provided a system for device interaction. The system includes means for detecting a triggering event and means for detecting a gaze, the means for detecting a gaze being activated when a trigger event is detected. The system also includes means for determining whether a gaze is detected, means for activating a display of the device when a gaze is detected, and means for deactivating the gaze detection after a predetermined amount of time if a gaze is not detected.” The gaze is the interaction, and a lack of gaze is a lack of interaction. Krulce [0026]: “If a gaze is not detected, gaze detection may be deactivated (208) until another triggering event is detected (202). In some embodiments, deactivating gaze detection may include powering down optical detection sensor module 122 and/or an associated sensor to conserve power. In some embodiments, deactivating gaze detection may include lowering a duty cycle or frame rate of the optical detection sensor so that fewer images are captured than when gaze detection is activated. In some embodiments, deactivating gaze detection may include the optical detection sensor continuing to capture images without optical detection sensor module 122 processing the captured images. In such embodiments, the captured images may be stored in a buffer, a memory internal to optical detection sensor module 122, or any of memories 108-112.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the detection of the gaze of a user to determine that the user is interacting with the system to determine a frame rate of a system as taught by Krulce with the system of Nixon in order adjust the frame rate of a graphics processing system based on aspects of the user’s presence. While they concern changing frame rates for different parts of the system, both Krulce and Nixon teach reducing the rate in response to the absence of a user interacting with the system closely.
As per claim 2, Nixon teaches the claimed:
2. The apparatus of claim 1, wherein the processing circuitry is further to: receive, from a proximity sensor, an indication that an object approaching an electronic device coupled to the controller the apparatus is not within a predetermined distance, and in response to the indication, to reduce a frame rendering rate of the graphics pipeline. (Nixon 3.1.2: “As accurately extracting pv for an arbitrary object in a video or animation is difficult in practice [16], the specific FPS required for pd=fd cannot be determined in real time on existing smartphones. However, Eq. (6) shows that regardless of underlying motion characteristics, FPS decreases as user-smartphone distance increases given the same source video or animation. As such, given the intended original viewing distance and framerate of a video or animation, Ddef and FPSdef, respectively, FPS can be reduced at larger viewing distances while maintaining TMRA so long as:” Nixon teaches a pre-determined distance. Nixon 3.1.2: “On modern smartphones, a Ddef of 12 inches can be considered as the default original viewing distance [9] [18], with FPSdef being equal to 60fps. Based on this, framerate may be reduced at viewing distances beyond Ddef while retaining the same level of perceived smoothness in a video or animation. Fig. 4b shows how framerate can be adjusted within the expected range of viewing distances for smartphone. Note that, due to existing hardware limitations, framerate cannot be increased beyond 60fps when the display is moved closer than Ddef. However, the figure shows that there exists a large opportunity to reduce framerate, and thus GPU workload and power consumption, without impacting visual quality within range of expected user-smartphone distances.”).
As per claims 22 and 29, these claims are similar in scope to limitations recited in claim 2, and thus are rejected under the same rationale.
As per claim 3, Nixon teaches the claimed:
3. The apparatus of claim 1, wherein the processing circuitry is further to: receive, from a proximity sensor, an indication that an object approaching the apparatus is within a predetermined distance, and in response to the indication, to activate a camera coupled to the apparatus. (Nixon 4.2.1: “where Fcam is the camera activation frequency; Pactcam is the power consumed by capturing a single image at a resolution of r; Palg+cam denotes the power consumption when the viewing distance is successfully calculated based on sufficient facial information provided by the captured image; Palg−cam denotes the power consumption of a failed distance calculation due to missing face; and a is the success ratio of the user's face being captured.”).
As per claims 23 and 30, these claims are similar in scope to limitations recited in claim 3, and thus are rejected under the same rationale.
As per claim 4, Nixon alone does not explicitly teach the claimed limitations.
However, Nixon in combination with Krulce teaches the claimed:
4. The apparatus of claim 3, wherein the processing circuitry is further to: determine whether an image input to the camera is a human, and in response to a determination that the image input is not a human, to reduce a frame rendering rate of the graphics pipeline. (Krulce [0031]: “FIG. 3 is a flowchart illustrating a process for activating display component 114 of device 100 using gaze detection, consistent with some embodiments. For the purpose of illustration, FIG. 3 will be described with reference to FIG. 1. The process 300 shown in FIG. 3 may be embodied in computer-readable instructions for execution by one or more processors in processing component 106 of device 100. In some embodiments, process 300 may be implemented by an operating system of device 100 stored in any of memories 108-112 and executed by processing component 106. In some embodiments, process 200 may be implemented as a background service in the operating system. As shown in FIG. 3, process 300 begins by initiating proximity detection using proximity detection sensor 124 (302). In some embodiments, proximity detection may be initiated by user 120. In some embodiments, proximity detection may be initiated automatically by the operating system of device 100. The initiation of proximity detection may be in response to a triggering event, such as described above with respect to FIG. 2. Proximity detection may also be initiated when device 100 is powered on, and may remain on for the duration that device 100 is powered on. Proximity detection sensor 124, in combination with processing component 106 and instructions stored in any of memories 108, 110, and 112, may determine if an object is detected in proximity of proximity detection sensor (304). According to some embodiments, determining if an object is in proximity may include detecting proximity using ultrasonic and/or infrared sensors. Moreover, determining if an object is in proximity may also include detecting a heat of an object so that non-user objects are not detected. Proximity detection may also include detecting a decrease in ambient light using an ambient light sensor caused by an object, such as user 120, occluding background light as user 120 comes into proximity of device 100.” The proximity detection accounts for non-user objects. These would be non-human objects. Krulce teaches that in the absence of a face, a reduced power mode is entered which includes a lower frame rate. Krulce [0032]: “…If a face is not detected, optical detection sensor module 122 and/or an associated sensor may enter a reduced power mode (310) until another object is detected in proximity of proximity detection sensor 124 (304). In some embodiments, a reduced power mode for optical detection sensor module 122 may include powering down an associated sensor to conserve power. In some embodiments, a reduced power mode may include lowering a duty cycle or frame rate of the associated sensor so that fewer images are captured than when optical detection sensor module 122 is activated. In some embodiments, a reduced power mode may include an optical sensor continuing to capture images without optical detection sensor module 122 processing the captured images. In such embodiments, the captured images may be stored in a buffer, a memory internal to optical detection sensor module 122, or any of memories 108-112.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the reduction in power based on the lack of presence of a user and detection of non-user objects as taught by Krulce with the system of Nixon to allow a reduced frame rate based on the presence of only non-human objects for a graphics pipeline to save power when a user is not directly interacting with it. While they concern changing frame rates for different parts of the system, both Krulce and Nixon teach reducing the rate in response to the absence of a user interacting with the system closely.
As per claims 24 and 31, these claims are similar in scope to limitations recited in claim 4, and thus are rejected under the same rationale.
As per claim 6, Nixon alone does not explicitly teach the claimed limitations.
However, Nixon in combination with Krulce teaches the claimed:
6. The apparatus of claim 3, wherein the processing circuitry is further to: determine whether an image input to the camera is a human, and in response to a determination that the image is a human: to activate at least one face-recognition module. (Nixon 4.2.1: “The power overhead of sampling from the smartphone's camera and determining a bounding box of the user's face, Pcam can be determined as
Pcam=Fcam∗{Pactcam+[α∗Palg+cam+(1−α)∗Palg−cam]}(13)
View Source
PNG
media_image1.png
20
24
media_image1.png
Greyscale
where Fcam is the camera activation frequency; Pactcam is the power consumed by capturing a single image at a resolution of r; Palg+cam denotes the power consumption when the viewing distance is successfully calculated based on sufficient facial information provided by the captured image; Palg−cam denotes the power consumption of a failed distance calculation due to missing face; and a is the success ratio of the user's face being captured.” Missing face indicates that whatever is in front of the camera is not human and thus the camera activation frequency does not get triggered. Once the camera is activated based on distance, it can be combined with the facial recognition of a user taught by Krulce. Krulce claims 16-18: “16. The method of claim 15, wherein determining whether a gaze is detected further comprises performing facial recognition on the detected face, wherein the display is activated when the detected face is recognized as an authorized user of the device.
17. The method of claim 9, wherein the gaze is detected and the display is activated, and wherein the method further comprises maintaining the display in an active state until the gaze is no longer detected.
18. The method of claim 9, wherein determining whether a gaze is detected further comprises performing facial recognition on a detected face, wherein a security challenge is bypassed when the detected face is recognized as an authorized user of the device.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the determination of an authorized user based on a proximate face as taught by Krulce with the system of Nixon in order to determine the presence of an authorized user to increase the frame rate of a graphics rendering system.
As per claims 26 and 33, these claims are similar in scope to limitations recited in claim 6, and thus are rejected under the same rationale.
Claims 5, 7, 25, 27, and 32 are rejected under 35 U.S.C. 103 as being unpatentable over Nixon in view of Krulce and further in view of Yi (Pub No. US 20160110590 A1).
As per claim 5, Nixon alone does not explicitly teach the claimed limitations.
However, Nixon in combination with Krulce and Yi teaches the claimed:
5. The apparatus of claim 4, wherein the processing circuitry is further to convert image data from the camera to a histogram;
And compare the histogram to preconfigured histogram data in a local memory; (Yi [0098]-[0099]: “[0098] The facial identification apparatus 100 creates a histogram for the test image converted to LBP features at step S305. During the process of creating a histogram, the facial identification apparatus 100 divides the converted test image into grid areas, creates a histogram for each divided grid area, and combines the histograms of each grid area thereby creating a single histogram feature. The histogram is composed of, for example, 256 bins, and the total number of grid areas is, for example, 49 by dividing the image by 7 on X axis and 7 on Y axis.
[0099] Thereafter, at step S307, a Chi-square distance is calculated for each pre-stored trained image and the test image wherein each image has been created as a histogram for each grid area.
where x and ξ represent a feature histogram to be compared, for example, x represents a histogram of a test image and ξ represents a histogram of a trained image. The index i represents one of the bins in the histogram, and j represents one specific grid area. For example, i ranges between 0 and 255, and j ranges between 0 and 48. w.sub.j represents a weight for a j-th grid area. While each grid area may have a different weight, a specific grid area with distinctive power for face recognition may have a weight greater than other grid areas. For example, the grid area for eyes or mouths may have a weight greater than other areas.”).
determine whether an image input to the camera is a face interacting with the display, and in response to a determination that the image is not a face interacting with the display, to reduce a frame rendering rate of the graphics pipeline; (Nixon 4.2.1: “where Fcam is the camera activation frequency; Pactcam is the power consumed by capturing a single image at a resolution of r; Palg+cam denotes the power consumption when the viewing distance is successfully calculated based on sufficient facial information provided by the captured image; Palg−cam denotes the power consumption of a failed distance calculation due to missing face; and a is the success ratio of the user's face being captured.” The power corresponds to a lower frame rate and correlates to the absence of a face. A larger viewing distance corresponds to a lower frame rate. Nixon 3.1.2: “As accurately extracting pv for an arbitrary object in a video or animation is difficult in practice [16], the specific FPS required for pd=fd cannot be determined in real time on existing smartphones. However, Eq. (6) shows that regardless of underlying motion characteristics, FPS decreases as user-smartphone distance increases given the same source video or animation. As such, given the intended original viewing distance and framerate of a video or animation, Ddef and FPSdef, respectively, FPS can be reduced at larger viewing distances while maintaining TMRA so long as:”).
determine whether an image input to the camera is a face interacting with the display, and in response to a determination that the image is a face interacting with the display, to maintain a full frame rendering rate of the graphics pipeline; (Krulce [0025]: “After a triggering event has been detected, gaze detection is activated (204). In some embodiments, activating gaze detection comprises activating or initiating a gaze detection module, function, and/or process, for example as executed by optical detection sensor module 122 and/or processing component 106, or a combination thereof. According to some embodiments, activating gaze detection includes activating optical detection sensor module 122 to capture and process, within the module 122 and/or by processing component 106, optical images for detecting a gaze. Moreover, activating gaze detection may include activating face detection and then, if a face is detected, activating gaze detection. In some embodiments, activating gaze detection may include increasing a duty cycle or frame rate of an optical detection sensor associated with module 122 such that optical detection sensor captures more images than before gaze detection was activated. Optical detection sensor module 122, using functionality included within module 122 and/or in combination with processing component 106 and instructions stored in any of memories 108, 110, and 112, may determine if a gaze is detected (206). A gaze may be detected using any of a number of gaze detection algorithms, and may include facial detection performed according to known facial detection algorithms. For example, if a face is detected, optical detection sensor module 122 may determine if the face is gazing in the direction of display component 114 by analyzing information captured by an optical detection sensor to determine a direction of the eyes on the detected face. Optical detection sensor module 122 may also determine if the face is gazing in the direction of display component 114 by analyzing other features of the detected face, such as corners of the eyes or mouth, eyebrows on the detected face, ears on the detected face, or other facial features that may be used to determine a direction of a gaze…”).
and determine whether an image input to the camera is a face of an authorized user, and in response to a determination that the image is not a face of an authorized user, to terminate frame rendering rate of the graphics pipeline. (Krulce [0041]: “If a gaze is detected by optical detection sensor module 122, a keylock or security challenge may be passed (418). In some embodiments, a detected gaze may be sufficient to bypass a keylock, but not sufficient to bypass a security challenge such as a password or PIN. According to some embodiments, optical detection sensor module 122 alone, or in combination with processing component 106, may be configured to detect a face and determine if the detected face matches a known face, such as a face of user 120, that may be stored in an internal memory of optical detection sensor module 122 or any of memories 108-112. If a detected face matches a face of user 120, a security challenge may be bypassed when display component 114 is activated. In some embodiments, a type of trigger may determine whether the security challenge is bypassed. For example, an alert of a newsfeed being updated may cause the security challenge to be bypassed, but reception of a personal email may still cause a security challenge to be presented. In some embodiments, authorized users are authorized for all content on the device. In other embodiments, each authorized user may have a set of permissions that determine if a security challenge will be presented upon recognition of the authorized user and/or if the user will be permitted access when a given type of triggering event occurs. In some embodiments, authenticating a user may comprise matching a color of the user's eyes to known eye colors and/or validating biometric data of the user, for example a shake of the user's eyes. Display component 114 may display content (420) after a user has passed security. The displayed content may be the triggering alert described above, such as a received e-mail, calendar alert, message and the like. The content may be displayed on display component for a predetermined time, after which optical detection sensor may determine if user 120 is still gazing at optical detection sensor module 122 (422). If a gaze is not detected, optical detection sensor module 122 may enter a reduced power mode (412), proximity sensor 124 may be deactivated (414), and display component 114 may be deactivated (416), such as described above, until another triggering event is detected (402) or user 120 manually activates any of these components. If a gaze is detected, display component 114 may remain activated (424) as long as a gaze is detected or until user manually deactivates display component 114, and proximity sensor 124 may be deactivated to save power (426). Moreover, a user 120 may be able to further interact with the content displayed on the activated display component 114 through gaze detection and/or eye tracking implemented by optical detection sensor module 122 alone or in combination with processing component 106, such as described above with respect to process 200 in FIG. 2.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the histogram to compare images of a face as taught by Yi with the system of Nixon in order to determine the presence of an authorized user to use that representation of data to compare input images to pretrained images to determine the distance and presence of faces in that image.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the determination of an authorized user based on a proximate face as taught by Krulce with the system of Nixon in order to determine the presence of an authorized user to increase the frame rate of a graphics rendering system.
As per claims 25 and 32, these claims are similar in scope to limitations recited in claim 5, and thus are rejected under the same rationale.
As per claim 7, Nixon alone does not explicitly teach the claimed limitations.
However, Nixon in combination with Yi teaches the claimed:
7. The apparatus of claim 6, wherein the processing circuitry is further tofurther comprising logic, at least partly including hardware logic, to: convert image data from the camera to a histogram; and compare the histogram to preconfigured histogram data in a local memory. (Yi [0098]-[0099]: “[0098] The facial identification apparatus 100 creates a histogram for the test image converted to LBP features at step S305. During the process of creating a histogram, the facial identification apparatus 100 divides the converted test image into grid areas, creates a histogram for each divided grid area, and combines the histograms of each grid area thereby creating a single histogram feature. The histogram is composed of, for example, 256 bins, and the total number of grid areas is, for example, 49 by dividing the image by 7 on X axis and 7 on Y axis.
[0099] Thereafter, at step S307, a Chi-square distance is calculated for each pre-stored trained image and the test image wherein each image has been created as a histogram for each grid area.
where x and ξ represent a feature histogram to be compared, for example, x represents a histogram of a test image and ξ represents a histogram of a trained image. The index i represents one of the bins in the histogram, and j represents one specific grid area. For example, i ranges between 0 and 255, and j ranges between 0 and 48. w.sub.j represents a weight for a j-th grid area. While each grid area may have a different weight, a specific grid area with distinctive power for face recognition may have a weight greater than other grid areas. For example, the grid area for eyes or mouths may have a weight greater than other areas.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the histogram to compare images of a face as taught by Yi with the system of Nixon in order to determine the presence of an authorized user to use that representation of data to compare input images to pretrained images to determine the distance and presence of faces in that image to change the framerate of a graphics pipeline taught by Nixon.
As per claim 27, this claim is similar in scope to limitations recited in claim 7, and thus are rejected under the same rationale.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THOMAS JOHN FOSTER whose telephone number is (571)272-5053. The examiner can normally be reached Mon, Fri 8:30-6. Tues-Thurs 7:30-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached at 571-272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THOMAS JOHN FOSTER/Examiner, Art Unit 2616
/HAI TAO SUN/Primary Examiner, Art Unit 2616
Prosecution Insights