Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2-7 and 11 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “nearby” in claim 2 is a relative term which renders the claim indefinite. The term “nearby” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The claim language is unclear as to which endpoint qualifies as being located “nearby” to the start point. For examination purposes, the claim limitation will be interpreted as “sets the area of interest based on a point on a line segment connecting a start point in the target detected area with an end point among end points in other ones of the detected areas.”
Claim 7 recites “wherein the processing circuitry treats a point on one short side in the target detected area as a start point, and treats a point on the other short side in each of the other detected areas as an end point”, which is indefinite. It is unclear what “the other short side in each of the other detected areas” is meant to refer to. For example, while it might be understood that for the rectangle target detected area, the side opposite the “one short side” would correspond to the other short side, it is unclear which side of the other detection areas would be considered the “other short side” as required by the claim. Specifically, is the other side in each other detected area the same side as that opposite to the “one short side” in the target detected area, or is the other short side meant to refer to the side opposite to a side within its own respective rectangle detected area. Thus, one of ordinary skill in the art would not be able to ascertain the scope of the claim. For examination purposes, the claim limitation will be interpreted as treating a point on either short side in each of the other detection areas as an end point.
The term “close” in claim 11 is a relative term which renders the claim indefinite. The term “close” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The claim language is unclear as to which light emitter qualifies as being located “close” to a center of the area of interest. For examination purposes, the claim limitation will be interpreted as requiring detection of a light emitter having a positional relationship to a center of the area of interest.
Claims 3-6 are rejected as being dependent on a rejected base claim.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 8, and 10-13 are rejected under 35 U.S.C. 101.
Claim 1 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of setting an area of interest based on image detection.
The claim recites: “An area setting device comprising processing circuitry to: detect an air outlet from image data captured of an indoor unit of an air conditioner, and set an area of interest based on a detected area of the air outlet that has been detected, the area of interest being a range in which a detection target is to be detected in the image data.”
The limitations, as drafted, are processes that, under their broadest reasonable interpretation, cover performance of the limitation in the human mind. A person can observe an image of an air conditioner, mentally identify the air outlets, and then mentally establish an area of interest for target detection. These steps correspond to cognitive processes that can be performed entirely in the human mind.
The judicial exception is not integrated into a practical application. For example, the claim recites the additional elements, “An area setting device comprising processing circuitry”. These additional elements are recited at a high level of generality such that they amount to generic computer components to perform generic computer functions. Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial expectation. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements are recited at a high-level of generality. It is therefore a judicial exception that is not integrated into a practical application, and does not include additional elements that are sufficient to amount to significantly more than the judicial exception. This claim is not patent eligible.
Claim 8 is rejected under 35 U.S.C. 101 because the claim recites additional elements which amount to merely implementing the abstract idea using a generic object detection model. Accordingly, this additional element does not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. This claim is not patent eligible.
Claim 10 is rejected under 35 U.S.C. 101 because the claimed invention is directed to a further limitation of the same abstract idea identified in the analysis of claim 1. For example, the person can mentally observe the set area of interest and detect light emitter which meets size requirements. This claim is not patent eligible.
Claim 11 is rejected under 35 U.S.C. 101 because the claimed invention is directed to a further limitation of the same abstract idea identified in the analysis of claim 1. For example, the person can mentally observe the set area of interest and detect light emitter which meets positional requirements. This claim is not patent eligible.
Claims 12 and 13 contain limitations found analogous to that of claim 1. Therefore, claims 12 and 13 are rejected for the same reason.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 12, and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Gyota (US 20190120517 A1) in view of Miyada et al. (US 20230394678 A1), (hereinafter Miyada).
Regarding claim 1, Gyota teaches an area setting device comprising processing circuitry (Gyota, see Figs. 2 and 3) to:
detect an air outlet from image data captured of an indoor unit of an air conditioner, and set an area of interest based on a detected area of the air outlet that has been detected (Gyota, pg. 4, paragraph 0066, “The air conditioner arrangement specifying unit 131 of the main server 130 specifies the position and orientation of an outlet in the image, based on the acquired arrangement specifying unit 131 specifies the position and orientation of the outlet by pattern matching with an appearance image of the air conditioner 150.”, pg. 4, paragraph 0069, “The image combining unit 136 of the main server 130 performs airflow analysis in consideration of interference between air blows and bending by a shield, based on the specified position and orientation of the outlet, the acquired air direction and volume of the air conditioner 150, and the position of the air conditioning affecting object transmitted from the user terminal 110 and stored (S17).”, pg. 5, paragraph 0082, “A plurality of LEDs may be arranged in the air conditioner 150 and the air conditioner arrangement specifying unit 131 may specify an outlet from the LED arrangement in the image. This can improve the accuracy of pattern matching. When, for example, the air conditioner 150 serves as an indoor unit equipped with outlets facing the four directions, the outlets can be accurately specified by diagonally arranging LEDs”, Air outlet position and orientations are specified in the image data. From this, air analysis can be performed, effectively establishing an area of interest for this purpose.).
Gyota does not teach the area of interest being a range in which a detection target is to be detected in the image data.
However, Miyada teaches the area of interest being a range in which a detection target is to be detected in the image data (Miyada, pg. 3, paragraphs 0032-0033, “The setting section 143 of the estimation section 14 sets an area of interest in which at least part of the detection target is included. The area of interest is an area in which at least part of the detection target is included, and is a noticed area that becomes a target of tracking hereinafter described. For example, the setting section 143 sets, for each of the joints of the person detected by the detection section 141, a square of a predetermined size centered at the joint as an area R of interest as depicted in FIG. 3… The tracking section 144 of the estimation section 14 tracks the detection target in the area R of interest set by the setting section 143, on the basis of the RGB image signal 113. The tracking section 144 may perform the tracking at a timing at which the RGB image signal 113 is generated (frame rate of the RGB camera 11) or may perform the tracking at a predetermined cycle or at a predetermined timing.”, see Fig. 3).
Gyota teaches detecting air outlets from image data to establish an area of interest for airflow analysis (Gyota, pg. 4, paragraph 0066 and pg. 4, paragraph 0069). Gyota does not teach setting an area of interest corresponding to a range in the image in which a target is to be detected. Miyada teaches setting an area of interest for a target to perform continuous tracking across a predetermined timing (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the air outlet detection of Gyota to include the area setting for continuous target tracking as taught by Miyada (Miyada, pg. 3, paragraphs 0032-0033). The motivation for doing so would have been observe changes in the air outlets over time, thereby improving the airflow analysis. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Gyota with Miyada to obtain the invention as specified in claim 1.
Claim 12 corresponds to claim 1, reciting an area setting method to execute the steps according to claim 1. Gyota in view of Miyada teaches an area setting method to execute the steps according to claim 1 (Gyota, see Figs. 5 and 9). As indicated in the analysis of claim 1, Gyota in view of Miyada teaches all the limitations according to claim 1. Therefore, claim 12 is rejected for the same reasons as claim 1.
Claim 13 corresponds to claim 1, additionally reciting a non-transitory computer readable medium storing an area setting program that causes a computer to function as an area setting device to execute the functions according to claim 1. Gyota in view of Miyada teaches the addition of a non-transitory computer readable medium storing an area setting program that causes a computer to function as an area setting device to execute the functions according to claim 1 (Gyota, pg. 2, paragraph 0041, lines 2-8, “The image capturing unit 112 can be implemented by the camera 502 controlled by the processor 506. The user operation unit 113 can be implemented by the input device 503 controlled by the processor 506. The image capturing range specifying unit 114 can be implemented by the processor 506 executing a program stored in the memory 507”). As indicated in the analysis of claim 1, Gyota in view of Miyada teaches all the limitations according to claim 1. Therefore, claim 13 is rejected for the same reasons as claim 1.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Gyota (US 20190120517 A1) in view of Miyada et al. (US 20230394678 A1) and further in view of Georgakis et al. (“Synthesizing Training Data for Object Detection in Indoor Scenes”, arXiv preprint arXiv:1702.07836, 2017), (hereinafter Georgakis).
Regarding claim 8, Gyota in view of Miyada teaches the area setting device according to claim 1. Gyota in view of Miyada does not teach wherein the processing circuitry detects an air outlet from image data captured of the indoor unit using an object detection model that takes, as input, image data and detects an air outlet, the object detection model being generated using, as input, learning data generated by performing at least one of resizing, rotating, and adjusting brightness on image data of an indoor unit.
However, Georgakis teaches wherein the processing circuitry detects an air outlet from image data captured of the indoor unit using an object detection model that takes, as input, image data and detects an air outlet, the object detection model being generated using, as input, learning data generated by performing at least one of resizing, rotating, and adjusting brightness on image data of an indoor unit (Georgakis, pg. 2, 2nd column, Section III. Approach, A. Synthetic Set Generations, paragraphs 1-2, “CNN-based object detectors require large amounts of annotated data for training, due to the large number of parameters that need to be learned. For object instance detection the training data should also cover the variations in the object’s viewpoint and other nuisance parameters such as lighting, occlusion and clutter… Our approach focuses on object instances and their superimposition into real scenes at different positions, scales, while reducing the difference in lighting conditions and exploiting proper context.”, pg. 3, 2nd column, 1st and 2nd full paragraphs, “The size of the object is determined by using the depth of the selected position and scaling the width w and height h accordingly: (see eqs.) where
z
-
is the median depth of the object’s training images, z is the depth at the selected position in the background image, and
w
^
,
h
^
are the scaled width and height respectively. The last step in our process is to blend the object with the background image in order to mitigate the effects of changes in illumination and contrast.”, pg. 4, section IV. Experiments, 1st paragraph, lines 1-9, “In order to evaluate the object detectors trained on composited images, we have conducted three sets of experiments on two publicly available datasets, the GMU-Kitchen Scenes [5] and the Washington RGB-D Scenes v2 dataset [11]. In the first experiment, training images are generated by choosing different compositing strategies to determine the effect of positioning, scaling, and blending on the performance. The object detectors are trained on composited images and evaluated on real scenes.”, see Figs. 1 and 2).
Gyota in view of Miyada teaches detecting air outlets of indoor air conditioners by using pattern matching (Gyota, pg. 4, paragraph 0069) and setting a detection area for continuous target detection (Miyada, pg. 3, paragraphs 0032-0033). Gyota in view of Miyada does not teach using an object detection model trained using augmented or transformed images, such as by being resized, rotated, or brightness adjusted. Georgakis teaches training and implementation of an object detection model which uses training data including composite images generated by performing resizing and illumination blending operations (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have replaced the pattern matching for air outlet detection of Gyota in view of Miyada with the object detection model as taught by Georgakis (Georgakis, pg. 2, 2nd column, Section III. Approach, A. Synthetic Set Generations, paragraphs 1-2, see Figs. 1 and 2). The motivation for doing so would have been to generate a data-driven model rather than apply a rule-based pattern matching, thereby improving accuracy of detection. Further, one skilled in the art could have substituted one known element for another, and the substitution would have yielded nothing more than predictable results.. Therefore, it would have been obvious to combine the teachings of Gyota in view of Miyada with Georgakis to obtain the invention as specified in claim 8.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Gyota (US 20190120517 A1) in view of Miyada et al. (US 20230394678 A1) and further in view of Georgakis et al. (“Synthesizing Training Data for Object Detection in Indoor Scenes”, arXiv preprint arXiv:1702.07836, 2017) and Arunmozhi et al. (US 20220065745 A1), (hereinafter Arunmozhi).
Regarding claim 9, Gyota in view of Miyada teaches the area setting device according to claim 1. Gyota in view of Miyada does not teach wherein the processing circuitry detects an air outlet from image data captured of the indoor unit using an object detection model that takes, as input, image data and detects an air outlet, the object detection model being generated using, as input, learning data including pieces of image data of an indoor unit.
However, Georgakis teaches wherein the processing circuitry detects an air outlet from image data captured of the indoor unit using an object detection model that takes, as input, image data and detects an air outlet, the object detection model being generated using, as input, learning data including pieces of image data of an indoor unit (Georgakis, pg. 2, 2nd column, Section III. Approach, A. Synthetic Set Generations, paragraphs 1-2, “CNN-based object detectors require large amounts of annotated data for training, due to the large number of parameters that need to be learned. For object instance detection the training data should also cover the variations in the object’s viewpoint and other nuisance parameters such as lighting, occlusion and clutter… Our approach focuses on object instances and their superimposition into real scenes at different positions, scales, while reducing the difference in lighting conditions and exploiting proper context.”, pg. 4, section IV. Experiments, 1st paragraph, lines 1-9, “In order to evaluate the object detectors trained on composited images, we have conducted three sets of experiments on two publicly available datasets, the GMU-Kitchen Scenes [5] and the Washington RGB-D Scenes v2 dataset [11]. In the first experiment, training images are generated by choosing different compositing strategies to determine the effect of positioning, scaling, and blending on the performance. The object detectors are trained on composited images and evaluated on real scenes.”, see Figs. 1 and 2).
Gyota in view of Miyada teaches detecting air outlets of indoor air conditioners by using pattern matching (Gyota, pg. 4, paragraph 0069) and setting a detection area for continuous target detection (Miyada, pg. 3, paragraphs 0032-0033). Gyota in view of Miyada does not teach using an object detection model trained on image data. Georgakis teaches training and implementation of an object detection model which uses training data including composite images generated by performing resizing and illumination blending operations (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have replaced the pattern matching for air outlet detection of Gyota in view of Miyada with the object detection model as taught by Georgakis (Georgakis, pg. 2, 2nd column, Section III. Approach, A. Synthetic Set Generations, paragraphs 1-2, see Figs. 1 and 2). The motivation for doing so would have been to generate a data-driven model rather than apply a rule-based pattern matching, thereby improving accuracy of detection. Further, one skilled in the art could have substituted one known element for another, and the substitution would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Gyota in view of Miyada with Georgakis to obtain the invention as specified above.
Gyota in view of Miyada and further in view of Georgakis does not teach the object detection model being generated using, as input, learning data including pieces of image data of an indoor unit in each of which an opening degree of the air outlet is different.
However, Arunmozhi teaches the object detection model being generated using, as input, learning data including pieces of image data of an indoor unit in each of which an opening degree of the air outlet is different (Arunmozhi, pgs. 4 and 5, paragraph 0057, “For example , the computer 42 can identify the spray pattern on the sensor window 58 based on the data received from the sensor 38 as a known type of spray pattern. For the purposes of this disclosure, "spray pattern” means an arrangement of the fluid output of the nozzle 40 onto the sensor window 58. The types of spray pattern can be a first spray pattern, a second spray pattern, and an absence of a spray pattern. The first spray pattern indicates that the valve 36 is in the partially open position, the second spray pattern indicates that the valve 36 is in the fully open position, and the absence of the spray pattern indicates that the valve 36 is in the fully closed position. For example, the computer 42 can identify the type of the spray pattern using conventional image-recognition techniques, e.g., a convolutional neural network programmed to accept images as input and output an identified type of obstruction… The convolutional neural network can be trained on images generated by the sensor 38 when the valve 36 is in the fully open, partially open, and fully closed positions.).
Gyota in view of Miyada and further in view of Georgakis teaches detecting air outlets using an object detection model trained using object images (Georgakis, pg. 2, 2nd column, Section III. Approach, A. Synthetic Set Generations, paragraphs 1-2, see Figs. 1 and 2), such as air outlets on an indoor unit. Gyota in view of Miyada and further in view of Georgakis does not teach the training images including the air outlet with different opening degrees. Arunmozhi teaches training a convolutional neural network using images corresponding to different opening degrees of a valve, such as fully open, partially open and fully closed (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified training images of Gyota in view of Miyada and further in view of Georgakis to include images of the valve in different opening degrees as taught by Arunmozhi (Arunmozhi, pgs. 4 and 5, paragraph 0057). The motivation for doing so would have been to learn different states of air outlets, thereby improving the accuracy of detection. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Gyota in view of Miyada and further in view of Georgakis with Arunmozhi to obtain the invention as specified in claim 9.
Claims 10 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Gyota (US 20190120517 A1) in view of Miyada et al. (US 20230394678 A1) and further in view of Yuan et al. (US 20220012465 A1), (hereinafter Yuan).
Regarding claim 10, Gyota in view of Miyada teaches the area setting device according to claim 1, wherein the processing circuitry detects a light emitter (Gyota, pg. 5, paragraph 0082, “A plurality of LEDs may be arranged in the air conditioner 150 and the air conditioner arrangement specifying unit 131 may specify an outlet from the LED arrangement in the image. This can improve the accuracy of pattern matching. When, for example, the air conditioner 150 serves as an indoor unit equipped with outlets facing the four directions, the outlets can be accurately specified by diagonally arranging LEDs.”).
Gyota in view of Miyada does not teach wherein the processing circuitry detects a light emitter whose size is smaller than a reference size from the area of interest that has been set.
However, Yuan teaches wherein the processing circuitry detects a light emitter whose size is smaller than a reference size from the area of interest that has been set (Yuan, pg. 4, paragraph 0028, “An object detection module 124 of application 118 attempts to identify/detect any individual objects that (1) are depicted in the well images received from visual inspection system 102 , and (2) could potentially be single cells . However, due to limitations of the model or algorithm employed by object detection module 124, and/or due to limitations in the resolution of the images processed by object detection module 124, object detection module 124 may be unable to ascertain, with sufficient accuracy or confidence, whether each detected object is in fact a single cell. Thus, each object detected by object detection module 124 is initially viewed only as a “ candidate ” for being a single cell. Object detection module 124 may detect objects using a relatively simple machine learning model, or using a non-machine learning algorithm (vision analysis software) that detects objects without requiring any training. In some embodiments, object detection module 124 utilizes OpenCV to process images , and to detect objects therein . In some embodiments, object detection module 124 only outputs objects that are above a minimum threshold size and / or below a maximum threshold size ( e.g. , threshold pixel widths or threshold numbers of pixels , etc. ), to avoid identifying objects that could not possibly be a single cell . For example, a minimum threshold size may filter out dead pixels in the image and very small contaminants , while a maximum threshold size may filter out very large contaminants or bubbles, etc. The operation of object detection module 124 is discussed in further detail herein.”, see also pg. 8, paragraph 0058)
Gyota in view of Miyada teaches detecting light emitters as part of air outlet detection from image data (Gyota, pg. 5, paragraph 0082). Gyota in view of Miyada does not teach applying size requirements for light emitter detection, such as detecting emitters smaller than a reference size. Yuan teaches applying minimum and maximum size thresholding to determine objects for detection (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the light emitter detection of Gyota in view of Miyada to include size thresholding as taught by Yuan (Yuan, pg. 4, paragraph 0028 and pg. 8, paragraph 0058). The motivation for doing so would have been to filter out irrelevant large objects, thereby improving detection accuracy. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Gyota in view of Miyada with Yuan to obtain the invention as specified in claim 10.
Regarding claim 11, Gyota in view of Miyada and further in view of Yuan teaches the area setting device according to claim 10, wherein the processing circuitry detects a light emitter located close to a center of the area of interest among light emitters whose size is smaller than the reference size (Miyada, pg. 3, paragraph 32, see Fig. 3, “For example, the setting section 143 sets each Of the joints or detected by the detection section a square of a predetermined size centered at the joint as an area R or interest as depicted in Fig. 3." Areas of interest centered around detected objects are set. The combination of Gyota in view of Miyada and further in view of Yuan would detect size-filtered light emitters centered at the area of interest.).
Allowable Subject Matter
Claims 2-7 are rejected under 35 U.S.C. 112(b) but would be allowable if base claim 2 were rewritten in independent form including all of the limitations of the base claim and any intervening claims and by overcoming the above rejection.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CONNOR LEVI HANSEN whose telephone number is (703)756-5533. The examiner can normally be reached Monday-Friday 9:00-5:00 (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CONNOR L HANSEN/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights