DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “the controller is configured to” in claims 1-10.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 10 and 20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claims 10 and 20, the claim limitation "sending, based on the pixel region not being detected, an inquiry for a depth with respect to pixels of pixel regions that are not determined as the transparent obstacle through the aligned depth image" (emphasis added) renders the claims indefinite because it is unclear what the purpose is for sending the inquiry for the depth information, how the inquiry is related to detection of the transparent obstacle, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-5, 11, and 13-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ozcan et al (IEEE Access 2022), hereinafter Ozcan in view of Wu (US 20220383513 A1), hereinafter Wu.
-Regarding claim 1, Ozcan discloses an apparatus for detecting a human behand a transparent obstacle, the apparatus comprising (Abstract; FIGS. 1-7; Page 66841, 2nd col., 2nd paragraph, “low-light and transparent obstacles …”; Page 66836, 2nd col, 2nd paragraph; Page 66838, 2nd col., 2nd paragraph): a red-green-blue-depth (RGB-Depth) camera configured to generate at least one image associated with the transparent obstacle, wherein the at least one image comprises red-green-blue (RGB) data and depth data (FIGS. 1-2, 4, 6; Tables 1, 4; Page 66834, 1st col., Sec. III., 3rd paragraph, “a Microsoft Kinect camera is used as an RGB-D sensor to obtain optical and depth data”; Page 66836, 2nd col, 2nd paragraph; Page 66838, 2nd col., 2nd paragraph); a thermal imaging camera configured to generate a thermal image associated with the transparent obstacle (FIGS. 1-2, 4, 6; Table 4; Page 66834, 1st col., Sec. III., 3rd paragraph, “Seek Compact Pro is used as a thermal sensor to detect thermal data in the environment”; Page 66836, 2nd col, 2nd paragraph; Page 66838, 2nd col., 2nd paragraph); and a controller coupled to the RGB-depth camera and the thermal imaging camera synchronized with each other, wherein the controller is configured to: align the at least one image and the thermal image (FIG. 2-5; Page 66832, 1st Col., 4th paragraph, “… using a real-time fusion, a new method establishing relationships between RGB-D and thermal sensors”, 2nd paragraph, Sec. II., 3rd paragraph, “different features are registered by using different sensors together”; Page 66834, Sec. III.); detect, based on the aligned at least one image and the aligned thermal image, a pixel region determined as the transparent obstacle by using an artificial intelligence model (FIGS. 4-7; Tables 4-6; Page 66832, 1st col., 1st paragraph, “Having all of this data simultaneously … the thermal and depth information of any desired object or region in the environment can be determined”; Page 66836, 2nd col., Sec. IV. A., 1st paragraph, “interest regions are assigned to overlap data from different sensors using the method described in Section III. … ROI images”; Page 66838, 2nd col., 2nd paragraph; Page 66841, 2nd paragraph – Page 66842, 1st paragraph, “… YOLOv4 neural network … increased even more with the training of the newly generated hybrid dataset …”); and estimate, based on the detected pixel region and based on the depth data of the aligned at least one image, a depth of the object associated with transparent obstacle (FIGS. 2-7).
Ozcan discloses an apparatus for detecting a human behand a transparent obstacle. However, Ozcan does not disclose detecting a transparent obstacle directly. A person of ordinary skills in the art would understand that the transparent obstacle can be determined based on detection process of a human behand the transparent obstacle.
In the same field of endeavor, Wu teaches a method for tracking a transparent object using machine learning (Wu: Abstract; FIGS. 1-2). Wu further teaches acquiring an environment capture camera or a visible light image data in real time and infrared thermal image data acquired in real time (Wu: FIG. 1, step 120; FIG. 2, blocks 121, 122) and providing the data to an object detection model using a convolutional neural network for transparent object detection and tracking (Wu: FIGS. 1-2). Wu also teaches that the environment capture camera is in frame synchronization with the infrared thermal imaging camera frame. (Wu: [0021]), and the transparent object can be detected and marked, and estimated depth values are assigned based transparent pixel identifier (Wu: [0062]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Ozcan with the teaching of Wu by detecting transparent obstacle with fusion of RGB-D image data and thermal image data in order to improve the performance for the detection of an object behind the transparent obstacle.
-Regarding claim 11, Ozcan discloses a method for detecting a human behand a transparent obstacle, the method comprising (Abstract; FIGS. 1-7; Page 66841, 2nd col., 2nd paragraph, “low-light and transparent obstacles …”; Page 66836, 2nd col, 2nd paragraph; Page 66838, 2nd col., 2nd paragraph): synchronizing a red-green-blue-depth (RGB-Depth) camera and a thermal imaging camera; aligning red-green-blue (RGB) image and a depth image generated (FIG. 2; Page 66832, 1st Col., 4th paragraph, “… using a real-time fusion, a new method establishing relationships between RGB-D and thermal sensors”, 2nd paragraph, Sec. II., 3rd paragraph, “different features are registered by using different sensors together”; Page 66834, Sec. III.) by the RGB-Depth camera with a thermal image generated by the thermal imaging camera (FIGS. 1-2, 4, 6; Tables 1, 4; Page 66834, 1st col., Sec. III., 3rd paragraph, “a Microsoft Kinect camera is used as an RGB-D sensor to obtain optical and depth data”; Page 66836, 2nd col, 2nd paragraph; Page 66838, 2nd col., 2nd paragraph), to generate an aligned RGB image, an aligned depth image, and an aligned thermal image, wherein each of the aligned RGB image, the aligned depth image, and the aligned thermal image is associated with the transparent obstacle (FIGS. 3-5); detecting, based on the aligned at least one image and the aligned thermal image, a pixel region determined as the transparent obstacle by using an artificial intelligence model (FIGS. 4-7; Tables 4-6; Page 66832, 1st col., 1st paragraph, “Having all of this data simultaneously … the thermal and depth information of any desired object or region in the environment can be determined”; Page 66836, 2nd col., Sec. IV. A., 1st paragraph, “interest regions are assigned to overlap data from different sensors using the method described in Section III. … ROI images”; Page 66838, 2nd col., 2nd paragraph; Page 66841, 2nd paragraph – Page 66842, 1st paragraph, “… YOLOv4 neural network … increased even more with the training of the newly generated hybrid dataset …”); and estimating, based on the detected pixel region and based on the depth data of the aligned at least one image, a depth of the object associated with transparent obstacle (FIGS. 2-7).
Ozcan discloses an apparatus for detecting a human behand a transparent obstacle. However, Ozcan does not disclose detecting a transparent obstacle directly. A person of ordinary skills in the art would understand that the transparent obstacle can be determined based on detection process of a human behand the transparent obstacle.
In the same field of endeavor, Wu teaches a method for tracking a transparent object using machine learning (Wu: Abstract; FIGS. 1-2). Wu further teaches acquiring an environment capture camera or a visible light image data in real time and infrared thermal image data acquired in real time (FIG. 1, step 120; FIG. 2, blocks 121, 122) and providing the data to an object detection model using a convolutional neural network for transparent object detection and tracking (FIGS. 1-2). Wu also teaches that the environment capture camera is in frame synchronization with the infrared thermal imaging camera frame. (Wu: [0021]), and the transparent object can be detected and marked, and estimated depth values are assigned based transparent pixel identifier (Wu: [0062]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Ozcan with the teaching of Wu by detecting transparent obstacle with fusion of RGB-D image data and thermal image data in order to improve the performance for the detection of an object behind the transparent obstacle.
-Regarding claims 3 and 13, Ozcan in view of Wu teaches the apparatus of claim 1 and the method of claim 11. The combination further teaches extracting an RGB camera parameter and a depth camera parameter from the RGB-Depth camera, through camera calibration; extracting a thermal imaging camera parameter from the thermal imaging camera; and by using the extracted RGB camera parameter, the extracted depth camera parameter, and the extracted thermal imaging camera parameter, generating the aligned RGB image, the aligned depth image, and the aligned thermal image (Ozcan: FIGS. 2-5).
-Regarding claims 4 and 14, Ozcan in view of Wu teaches the apparatus of claim 1 and the method of claim 11. The combination further teaches wherein the aligning the RGB image and the depth image with the thermal image comprises: aligning the depth image with the RGB image; and aligning the aligned depth image and the aligned RGB image with the thermal image, to generate the aligned RGB image, the aligned depth image, and the aligned thermal image (Ozcan: FIGS. 2-5).
-Regarding claims 5 and 15, Ozcan in view of Wu teaches the apparatus of claim 1 and the method of claim 11. The combination further teaches wherein the detecting the pixel region determined as the transparent obstacle comprises: extracting, via a deep learning model, a feature of each of the aligned RGB image and the aligned thermal image; extracting, based on the extracted feature, regions where different images are formed as the pixel region; and detecting pixels of the pixel region as the transparent obstacle (Ozcan: FIGS. 2-7; Page 66834, 2nd col., 2nd paragraph, “The differences in distance and slope between the detected calibration points are calculated and compared to each other”; Page 66836, 2nd col., Sec. IV. A., 1st paragraph, “interest regions are assigned to overlap data from different sensors using the method described in Section III. … ROI images”; Page 66838, 1st col., 2nd paragraph, “… detection and classification are measured using a pre-trained convolutional neural network …”; Page 66841, 2nd paragraph – Page 66842, 1st paragraph, “… YOLOv4 neural network … increased even more with the training of the newly generated hybrid dataset …”; See also Wu: FIGS. 1-2)
Claim(s) 2 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ozcan et al (IEEE Access 2022), hereinafter Ozcan in view of Wu (US 20220383513 A1), hereinafter Wu, and further in view of Jiang et al (arXiv:2304.00157v2 2023), hereinafter Jiang.
-Regarding claims 2 and 12, Ozcan in view of Wu teaches the apparatus of claim 1 and the method of claim 12.
Ozcan in view of Wu does not teach estimating depth of the transparent obstacle for a robot navigation with without colliding with the transparent obstacle.
However, Jiang is an analogous art pertinent to the problem to be solved in this application and teaches a method for perceiving transparent objects to enables robots achieving higher levels of autonomy (Jiang: Abstract; FIGS. 1-11). Jiang further teaches comparing the estimated depth of the transparent obstacle with a collision range of a robot; controlling, based on the estimated depth being within the collision range, the robot to stop; and controlling, based on the estimated depth being out of the collision range, the robot to normally drive (Jiang: Page 2, 1st col., 1st paragraph, “(1) locating transparent objects and (2) accurately estimating the depth of transparent object …”; Page 6, Sec. III., 1st paragraph, “It allows autonomous robots to navigate in unknown environments … without colliding with glass walls or windows”; Page 7, 2nd col., 4th paragraph, “… depth information of transparent objects … self-localization”; Table 1).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify the teaching of Ozcan in view of Wu with the teaching of Wu by provide robotic perception of transparent objects in order to provide a real-world application.
Allowable Subject Matter
Claims 6-9 and 16-19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Huo et al (arXiv:2204.05453v4 16 Mar 2023), hereinafter Huo teaches a method for glass segmentation utilizing paired RGB and thermal images.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIAO LIU whose telephone number is (571)272-4539. The examiner can normally be reached Monday-Thursday and Alternate Fridays 8:30-4:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format.
For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XIAO LIU/Primary Examiner, Art Unit 2664
Prosecution Insights