DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-20 are pending.
Priority
This application claims the priority benefit under 35 U.S.C. § 119(e) of U.S. Provisional Application No. 63/582,052, filed on September 12, 2023.
Information Disclosure Statement
The IDS filed 09/17/24 has been considered.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-2 and 11-12 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by IDZIK et al. (US 20150029085 A1 Hereinafter “IDZIK”).
Regarding claim 1, IDZIK teaches a method for performing gesture recognition, the
method comprising:
detecting a gesture using a primary modality ([0025]: “This portable electronic device also includes a plurality of 3D gesture sensors including at least a first 3D gesture sensor 152 and a second 3D gesture sensor 153”. The first 3D gesture sensor acts as the primary modality. It captures the first portion of gestures as seen in Fig. 3 “In this example a camera sensor 304 serves to detect the first portion 303 of the 3D gesture 302” [0038]. It differs from the second and subsequent 3D gesture sensors by being different modalities “At least the first and second 3D gesture sensors 152 and 153 differ from one another as regards their gesture-sensing modalities. Any additional 3D gesture sensors 154, when provided, can employ a gesture-sensing modality that is similar or identical to that of either or both of the first and second 3D gesture sensors 152 and 153 or, if desired, different yet again” [0026]);
evaluating an expected accuracy gain (EAG) to identify a modality that yields a maximum relative EAG among the primary modality and one or more secondary modalities ([0013]: “By one approach, for example, the control circuit employs both 3D gesture sensors in a temporally-overlapping manner to reliably and accurately detect the 3D gesture. In this case, for example, the detection weaknesses of one gesture-sensing modality can be compensated for by the well-chosen strengths of the other gesture-sensing modality. So configured, a given device may be able to reliably accommodate a wider range of 3D gestures than has previously been considered practical or possible”. For the second modality to be chosen, the expected accuracy gain of using the second modality would have to be evaluated to know that it covers for the weakness of the first modality leading to a maximum relative expected accuracy gain relative to the first modality. Weakness for detecting gestures means lower accuracy, so compensation for detection weakness using a second modality means it is expected to have higher accuracy of gesture recognition); and
activating the one or more secondary modalities for detecting the gesture if the one or more secondary modalities correspond to the modality that yields the maximum relative EAG ([0013]: “By one approach, for example, the control circuit employs both 3D gesture sensors in a temporally-overlapping manner to reliably and accurately detect the 3D gesture. In this case, for example, the detection weaknesses of one gesture-sensing modality can be compensated for by the well-chosen strengths of the other gesture-sensing modality. So configured, a given device may be able to reliably accommodate a wider range of 3D gestures than has previously been considered practical or possible”. The second modality is selected based on if it is believed that the second modality will cover for the weakness in the first gesture sensing modality, hence presuming a maximum relative expected accuracy gain relative to the first modality. The result of determining there is expected accuracy gain leads to activation of the second modality being activated in a temporarily overlapping manner).
Regarding claim 2, IDZIK teaches the method of Claim 1, further comprising:
detecting a first portion of the gesture for a duration that is less than a duration of an entire length of the gesture (Fig. 3, [0038]: “In FIG. 3, the user has begun a particular 3D gesture 302 by moving their extended index finger 301 through the air. In this example a camera sensor 304 serves to detect the first portion 303 of the 3D gesture 302”. The first portion of the gesture means that it is less than the full duration of the gesture, this is further supported by a second and third part shown inf Figs. 4-5).
Regarding claim 11, the content of claim 11 is similar to the content of claim 1, with the additional teachings of a processor and memory. IDZIK also discloses this information ([0023]: “The portable electronic device includes an operating system 146 and software programs, applications, or components 148 that are executed by the control circuit 102 and are typically stored in a persistent, updatable store such as the memory 110”). Therefore, claim 11 is rejected for the same reasons of anticipation as claim 1, along with the additional teachings above.
Regarding claim 12, the content of claim 2 is similar to the content of claim 1, therefore it is rejected for the same reasons of anticipation as claim 1.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over IDZIK et al. (US 20150029085 A1 Hereinafter “IDZIK”) in view of MARCHENKO et al. (US 20180329501 A1 Hereinafter “MARCHENKO”).
Regarding claim 3, IDZIK teaches the method of Claim 1, further comprising:
([0013]: “By one approach, for example, the control circuit employs both 3D gesture sensors in a temporally-overlapping manner to reliably and accurately detect the 3D gesture. In this case, for example, the detection weaknesses of one gesture-sensing modality can be compensated for by the well-chosen strengths of the other gesture-sensing modality. So configured, a given device may be able to reliably accommodate a wider range of 3D gestures than has previously been considered practical or possible”. For the second modality to be chosen, the expected accuracy gain of using the second modality would have to be evaluated to know that it covers for the weakness of the first modality leading to a maximum relative expected accuracy gain relative to the first modality. Weakness for detecting gestures means lower accuracy, so compensation for detection weakness using a second modality means it is expected to have higher accuracy of gesture recognition).
IDZIK does not expressly disclose determining a probe classifier according to a confidence score for a set of gesture classes, and determining the EAG based on the probe classifier.
However, MARCHENKO teaches determining a probe classifier according to a confidence score for a set of gesture classes ([0096]: “The electronic device operating in the extended mode may use a preliminary estimation algorithm in order to identify a gesture type intended by the movement of the object. According to the preliminary estimation algorithm, the electronic device can compare at least one received image corresponding to a moving object with a pre-stored image and, according to the comparison result, identify whether the gesture type is a simple gesture type or a complex gesture type”. The preliminary estimation algorithm acts as the probe classifier, it detects whether the current gesture is simple or complex, to be able to make that deduction, it would have to have a confidence metric (such as a score) associated with the set of gesture classes (simple and complex)), and determining the EAG based on the probe classifier ([0096]: “The electronic device operating in the extended mode may use a preliminary estimation algorithm in order to identify a gesture type intended by the movement of the object. According to the preliminary estimation algorithm, the electronic device can compare at least one received image corresponding to a moving object with a pre-stored image and, according to the comparison result, identify whether the gesture type is a simple gesture type or a complex gesture type”. By determination of a complex or simple gesture, an expected accuracy gain is calculated, this is supported by the fact that the detection switches from “extended mode” to “full performance mode” due to expectation that full performance mode will result in more accurate motion detection due to the complex motion).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify IDZIK’s gesture detection system to include MARCHENKO’s use of an initial classifier to analyze a detected gesture to determine expected accuracy gain because such a modification is the result of applying a known technique to a known device ready for improvement to yield predictable results. More specifically, MARCHENKO’s use of an initial classifier to analyze a detected gesture to determine expected accuracy gain permits analysis of the gesture to determine the most effective method of data capture. This known benefit in MARCHENKO is applicable to IDZIK’s gesture detection system as they both share characteristics and capabilities, namely, they are directed to altering data capture modalities for improved gesture detection. Therefore, it would have been recognized that modifying IDZIK’s gesture detection system to include MARCHENKO’s use of an initial classifier to analyze a detected gesture to determine expected accuracy gain would have yielded predictable results because (i) the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate RCHENKO’s use of an initial classifier to analyze a detected gesture to determine expected accuracy gain in altering data capture modalities for improved gesture detection and (ii) the benefits of such a combination would have been recognized by those of ordinary skill in the art.
Regarding claim 13, the content of claim 13 is similar to the content of claim 3, therefore it is rejected for the same reasons of obviousness as claim 13.
Claims 6, 8, 16, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over IDZIK et al. (US 20150029085 A1 Hereinafter “IDZIK”) in view of KARAGIANNIS et al. (US 20230033951 A1 Hereinafter “KARAGIANNIS”).
Regarding claim 6, IDZIK teaches the method of Claim 1, further comprising:
deactivating a first type of sensor in response to the EAG ([0040]: “Both the camera sensor 304 and the ultrasonic sensor 501 will serve well to reliably and accurately distinguish amongst shapes in these regards. In this case, however, the ultrasonic sensor 501 consumes less energy than the camera sensor 304 and therefore the control circuit 102 employs the ultrasonic sensor 501 as part of an overall power-conserving strategy while still assuring accurate detection of the user's 3D gestures”. A first type of sensor (capacitive) sensor is deactivated when the ultrasonic sensor is determined to be used at it has a EAG similar to the camera which is good enough to use for gesture detection)
IDZIK does not expressly disclose deactivating a sensor relative to a threshold of accuracy.
However, KARAGIANNIS teaches deactivation of sensors that are below confidence thresholds ([0024]: “The operations then deactivate 604 the sensors within the set of sensors having confidence scores that do not satisfy the defined threshold, e.g., being less than the defined threshold value”).
At the time the invention was effectively filed, it would have been obvious to one of ordinary skill in the art to modify IDZIK’s sensor system for gesture detection to include KARAGIANNIS’s deactivating of sensors relative to an accuracy threshold because such a modification is taught, suggested, or motivated by the art. More specifically, the motivation to modify IDZIK to include KARAGIANNIS is expressly provided by KARAGIANNIS, stating that deactivating of poor performing sensors will save power ([0022]: “The system deactivates all non-selected sensors to reduce the total power consumption and other system resource utilization (e.g., processor utilization, memory utilization, network utilization, etc.)”). This improves overall power efficiency for any multi-sensor system. Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify IDZIK’s sensor system for gesture detection to include KARAGIANNIS’s deactivating of sensors relative to an accuracy threshold with the motivation of improving power efficiency. The person of ordinary skill in the art would have recognized the benefit of improved power efficiency.
Regarding claim 8, IDZIK teaches the method of Claim 1, further comprising:
detecting the gesture for a set of frames([0025]: “This portable electronic device also includes a plurality of 3D gesture sensors including at least a first 3D gesture sensor 152 and a second 3D gesture sensor 153”. The first 3D gesture sensor acts as the primary modality. It captures the first portion of gestures as seen in Fig. 3 “In this example a camera sensor 304 serves to detect the first portion 303 of the 3D gesture 302” [0038]. It differs from the second and subsequent 3D gesture sensors by being different modalities “At least the first and second 3D gesture sensors 152 and 153 differ from one another as regards their gesture-sensing modalities. Any additional 3D gesture sensors 154, when provided, can employ a gesture-sensing modality that is similar or identical to that of either or both of the first and second 3D gesture sensors 152 and 153 or, if desired, different yet again” [0026]. Since there is multiple gesture portions as seen in Fig. 3, it means multiple image/frames are needed, Fig. 34 shows a set of frames being analyzed); and
IDZIK does not expressly disclose determining whether the EAG for a subsequent non-overlapping set of frames is greater than or equal to a predefined threshold.
However, KARAGIANNIS teaches determining whether the EAG for a subsequent non-overlapping set of frames is greater than or equal to a predefined threshold ([0027]: “As the vehicle travels along the path 209 it performs various operations discussed in embodiments of the present disclosure. For example, in a first region 205 the first sensor 201 is determined by the computing device 200 to satisfy a defined rule which causes activation or keeping the first sensor 201 activated while the vehicle remains in the first region 205. The vehicle travels into a second region 207 where the computing device 200 determines that the first sensor 201 delivers a confidence score that does not satisfy a defined threshold, e.g., less than a defined threshold. Two alternative operational embodiments can be triggered by the computing device 200. In one embodiment, the unsatisfactory confidence score of the first sensor 201 causes all of the onboard sensors that were previously deactivated (in this case only the second sensor 203) to be reactivated and analyzed to determine which of the reactivated sensors yields confidence scores that satisfy the defined threshold”. For the vehicle to activate and deactivate sensors, an expected accuracy gain (confidence) must be calculated. Low confidence in the first sensor and increased confidence in the second sensor would be an increased expected accuracy gain (expecting the accuracy to be higher with the second sensor. This must be calculated in subsequent frames (frames captures in the second area are subsequent to frames captured in the first area). This is relative to a threshold to determine if sensors are activated and deactivated).
At the time the invention was effectively filed, it would have been obvious to one of ordinary skill in the art to modify IDZIK’s sensor system for gesture detection to include KARAGIANNIS’s determination of sensor accuracy relative to a threshold in subsequent frames because such a modification is taught, suggested, or motivated by the art. More specifically, the motivation to modify IDZIK to include KARAGIANNIS is implicitly provided by KARAGIANNIS, stating that analysis of sensor over subsequent frames leads to overall save in power expenditure and increase in accuracy of detection ([0055]: “This way, in future sessions of the same or a different operations of a device, if the performance of the currently activated sensors degrades, the operations will directly activate the most suitable set of sensors by retrieving their overall confidence score Λ.sub.ξ from the grid the device is moving towards, and will accordingly deactivate the poorly performing sensors. This will save power since there is no need to activate the entire set of the on-board sensors for online evaluation”). This improves overall power efficiency for any multi-sensor system and increase accuracy by determining for subsequent frames if there is a more suitable set of sensors to be using. Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify IDZIK’s sensor system for gesture detection to include KARAGIANNIS’s determination of sensor accuracy relative to a threshold in subsequent frames with the motivation of improving power efficiency and detection accuracy. The person of ordinary skill in the art would have recognized the benefit of improved power efficiency and detection accuracy.
Regarding claim 16, the content of claim 16 is similar to the content of claim 6, therefore it is rejected for the same reasons of obviousness as claim 6.
Regarding claim 18, the content of claim 18 is similar to the content of claim 8, therefore it is rejected for the same reasons of obviousness as claim 8.
Claims 7 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over IDZIK et al. (US 20150029085 A1 Hereinafter “IDZIK”) in view of Liu et al. (“LD-ConGR: AL arge RGB-D Video Dataset for Long-Distance Continuous Gesture Recognition” Hereinafter “Liu”).
Regarding claim 7, IDZIK teaches the method of Claim 1, wherein detecting the gesture ([0025]: “This portable electronic device also includes a plurality of 3D gesture sensors including at least a first 3D gesture sensor 152 and a second 3D gesture sensor 153”. The first 3D gesture sensor acts as the primary modality. It captures the first portion of gestures as seen in Fig. 3 “In this example a camera sensor 304 serves to detect the first portion 303 of the 3D gesture 302” [0038]. It differs from the second and subsequent 3D gesture sensors by being different modalities “At least the first and second 3D gesture sensors 152 and 153 differ from one another as regards their gesture-sensing modalities. Any additional 3D gesture sensors 154, when provided, can employ a gesture-sensing modality that is similar or identical to that of either or both of the first and second 3D gesture sensors 152 and 153 or, if desired, different yet again” [0026]).
IDZIK does not expressly disclose decreasing a frame rate as a time duration of detecting the gesture increases.
However, Liu teaches decreasing a frame rate as a time duration of detecting the gesture increases (Page 3311, Table 5: Table 5 shows the key frame rate with sampling compared to the normal frame rate for gesture detection. For palm they reduce 10.96 frames of average duration to 6.85 frames, effectively reducing the frames by 0.375 frames. For the fist they reduce the rate of frames from 16.38 frames of average duration to 8.86 frames, effectively recuing the rate of frames by 0.459 frames. This trend continues with the highest frame rate (Shift right) being reduced the most (0.589 frames). So the longer the time duration of the gesture the more the gesture’s frame rate gets reduced).
At the time the invention was effectively filed, it would have been obvious to one of ordinary skill in the art to modify IDZIK’s sensor system for gesture detection to include Liu’s adaptive frame rate because such a modification is taught, suggested, or motivated by the art. More specifically, the motivation to modify IDZIK to include Liu is expressly provided by Liu, stating that reducing frame rate in this way reduces the impact of gesture speed and duration while maintaining high speed and accurate gesture recognition with fewer frames ([0022]: “In view of this, we try to extract key frames of the video based on inter-frame difference to remove redundant frames for long-duration gestures. Results show that the key frame sampling strategy reduces the impact of gesture speed and duration, and realizes high-speed and accurate recognition with fewer frames”). Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify IDZIK’s sensor system for gesture detection to include Liu’s adaptive frame rate with the motivation of reducing the impact of gesture speed and duration while maintaining high speed and accurate gesture recognition for fewer frames. The person of ordinary skill in the art would have recognized the benefit of reduced impact of gesture speed and duration while maintaining high speed and accurate gesture recognition for fewer frames.
Regarding claim 17, the content of claim 17 is similar to the content of claim 7, therefore it is rejected for the same reasons of obviousness as claim 7.
Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over IDZIK et al. (US 20150029085 A1 Hereinafter “IDZIK”) in view of SONG et al. (US 20220309686 A1 Hereinafter “SONG”).
Regarding claim 9, IDZIK teaches the method of Claim 1, further comprising:
IDZIK does not expressly disclose updating a channel map based on a weighted sum of channel maps.
However, SONG teaches updating a channel map based on a weighted sum of channel maps ([0012]: “The updating of the refined template feature map may include: determining a new feature map based on a search feature map of the search image; and updating the refined template feature map by determining a weighted sum of the template feature map and the new feature map”. The feature maps are functionally similar to channel maps, and they use a weighted sum to update a refined template map).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify IDZIK’s gesture detection system to include SONG’s updating of a channel map using a weighted summation of channel maps because such a modification is the result of applying a known technique to a known device ready for improvement to yield predictable results. More specifically, SONG’s updating of a channel map using a weighted summation of channel maps permits accurate updating of channel maps by using a weighted summation of the other channel maps, allowing for more accurate updating. This known benefit in SONG is applicable to IDZIK’s gesture detection system as they both share characteristics and capabilities, namely, they are directed to using images and image analysis to detect objects. Therefore, it would have been recognized that modifying IDZIK’s gesture detection system to include SONG’s updating of a channel map using a weighted summation of channel maps would have yielded predictable results because (i) the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate SONG’s updating of a channel map using a weighted summation of channel maps in using images and image analysis to detect objects and (ii) the benefits of such a combination would have been recognized by those of ordinary skill in the art.
Regarding claim 19, the content of claim 19 is similar to the content of claim 9, therefore it is rejected for the same reasons of obviousness as claim 9.
Claims 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over IDZIK et al. (US 20150029085 A1 Hereinafter “IDZIK”) in view of YIN et al. (US 20230334632 A1 Hereinafter “YIN”).
Regarding claim 10, IDZIK teaches the method of Claim 1, further comprising:
IDZIK does not expressly disclose updating a channel map based on a weighted probability of channel maps.
However, YIN teaches updating a channel map based on a weighted probability of channel maps ([0011]: “determining importance of each input channel of the current layer according to a feature map of each sample data in the sample image data set at each input channel of the current layer; [0013] setting an importance weighting coefficient for each input channel according to the importance of each input channel; [0014] determining a sampling probability of each input channel by calculating a weighted importance function of each input channel and a sum function; [0015] performing multiple rounds of sampling on a corresponding input channel according to the sampling probability of each input channel, performing multiple sampling on an input channel set of the current layer per round according to the sampling probability to obtain a kernel set, calculating and accumulating feature map reconstruction errors corresponding to a channel kernel set, and obtaining an updated value of a convolution kernel weight of the current layer by calculating an optimization function that minimizes feature map reconstruction errors”. The kernels act as channel maps, and they are generated based on the sampling probability of each input channel (which acts as a weighted probability since they must have some weight). These kernels are updated for minimizing reconstruction error. These updated kernels are then used to generate feature maps which are updated version of the previous feature maps. So the channel maps (feature maps) are updated (via generation of updated kernels) based on the weighted probability of channel maps (the kernels are generated based on the probability sampling of input channels which must have some weight).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify IDZIK’s gesture detection system to include YIN’s updating of a channel map using a weighted probability of channel maps because such a modification is the result of applying a known technique to a known device ready for improvement to yield predictable results. More specifically, YIN’s updating of a channel map using a weighted probability of channel maps permits accurate updating of channel maps by using a weighted probability of the other channel maps, allowing for more accurate updating by minimizing reconstruction error. This known benefit in YIN is applicable to IDZIK’s gesture detection system as they both share characteristics and capabilities, namely, they are directed to using images and image analysis to detect objects. Therefore, it would have been recognized that modifying IDZIK’s gesture detection system to include SONG’s updating of a channel map using a weighted summation of channel maps would have yielded predictable results because (i) the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate YIN’s updating of a channel map using a weighted probability of channel maps in using images and image analysis to detect objects and (ii) the benefits of such a combination would have been recognized by those of ordinary skill in the art.
Regarding claim 20, the content of claim 20 is similar to the content of claim 10, therefore it is rejected for the same reasons of obviousness as claim 10.
Allowable Subject Matter
Claims 4-5 and 14-15 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Yarlagadda et al. (US 11017513 B1) teaches selecting sensor based off detection hypothesis for accurate object detection.
UCHIYAMA et al. (US 20220343757 A1) teaches changing sensor type being used for detection.
Schwarz et al. (US 11768544 B2) teaches two model gesture recognition
Any inquiry concerning this communication or earlier communications from the examiner should be directed to STEFANO A DARDANO whose telephone number is (703)756-4543. The examiner can normally be reached Monday - Friday 11:00 - 7:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Greg Morse can be reached at (571) 272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/STEFANO ANTHONY DARDANO/ Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights