Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement filed 19 September 2025 fails to comply with 37 CFR 1.97(c) because it lacks the timing fee set forth in 37 CFR 1.17(p). It has been placed in the application file, but the information referred to therein has not been considered.
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Response to Amendment
In response to the amendments to claims 1-7, 9, and 13, and the cancelation of claim 8, the rejection of claims 1-7, 9, and 13 under 35 § U.S.C. 112(b) are withdrawn.
Response to Arguments
Applicant’s arguments, filed 26 November 2025, have been fully considered but they are not persuasive. Applicant’s remarks and arguments, as best understood by Examiner, are paraphrased below:
Criminisi fails to disclose, suggest, or teach identification of a background object from a plurality of background objects based on depth information.
Furthermore, Examiner’s reliance on paras. 0066 and 0076-0077, relied on for disclosing the background element and identifying a smallest value as being a background element on top of which the user is.
Respectfully, Examiner disagrees.
Regarding both arguments, Examiner contends that, by the broadest reasonable interpretation of the claim language of both independent claim 1 and dependent claim 8, the disclosure of Criminisi does, in fact, disclose all of the limitations of both original claim 1 and amended claim 1. In response to Applicant’s assertions (the entire rejection follows below), Examiner has included specific references to Criminisi’s teachings here:
“[I]dentify a background object where the object of interest is located, from among the plurality of background objects, based on a difference between the depth information of the object of interest and the depth information of each of the plurality of background objects”.
Having reviewed the Specification of the instant application, specifically para. 0051, Examiner is interpreting “background objects” to be any objects that are not the object of interest within the frame of the indoor image. Thus, the “foreground [image] objects” of Criminisi et al., which refer to non-object of interest image objects which are located in proximity to the object of interest and are not segmented out by the method of Criminisi, are “background objects” under the definition of the instant application as interpreted by Examiner. As a result, the disclosure of paras. 0066-0069, altogether disclosing the removal of a background but the inclusion of foreground objects, would constitute identification of individual background objects from a set of background objects. The depth calculations of any of these individual background objects are disclosed using geodesic distances based on depth images within paras. 0076-0081, satisfying the depth information limitation of the claim. Finally, paras. 0090-0093 disclose mask creation for thresholding foreground image objects and the key object of interest, proving the ability of geodesic distance metric to accurately segment pertinent image objects using depth.
“[W]herein the processor is further configured to: identify a smallest value from among differences between the depth information of the object of interest and the depth information of each of the plurality of background objects; and identify a background object corresponding to the smallest value as a background object on top of which the object of interest is located”
Using the aforementioned interpretation for the claim term “background [image] object[(s)]”, Examiner further maintains that the limitations of former claim 8 (incorporated within newly-amended claim 1) is also taught by Criminisi. Specifically, the depth calculations of any of the individual background objects are disclosed using geodesic distances based on depth images within paras. 0076-0081, satisfying the depth information limitation of the claim. In addition, Criminisi discloses generation of geodesic distance images and subsequent subtraction and thresholding to determine key foreground objects and mitigate image bleeding for output masks. Although Examiner maintains that one ordinarily skilled in the art would be able to identify that the smallest observed distance determined from the subtracted, post-threshold geodesic image would necessarily be the closest depthwise image object to the user/object of interest. Nonetheless, Examiner further directs Applicant to paras. 0130-0132, wherein the user/object of interest is sitting on a chair which, utilizing geodesic segmentation, is determined to be a separate background object upon which the user is sitting (smallest geodesic distance from the user) using obtained geodesic distances and an image object classifier.
Therefore, Criminisi fully teaches all limitations of newly amended claim 1, including the limitations of former claim 8. Thus, the rejection under 35 U.S.C. § 102(a)(1) is maintained.
Claim Objections
Claims 1 is objected to because of the following informalities:
The phrase “a processor, when executing the instructions, is configured to…” is grammatically incorrect. The phrase should read “and a processor which, when executing the instructions, is configured to…”.
Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 6, 10, and 15 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Criminisi et al. (US PG Pub 20140126821, hereinafter “Criminisi”).
Regarding claim 1, Criminisi discloses an electronic apparatus comprising:
a camera (para. 0039-0040 and 0044, wherein the capture device is a depth camera configured to take pictures of the scene, and in the observed embodiment, an RGB-D camera);
a memory storing instructions (para. 0149); and
and a processor, when executing the instructions (para. 0044, where the processor is configured to be in communication with the image sensor) is configured to: identify a first area of a threshold size in an image obtained by the camera, the first area including an object of interest (paras. 0034 and 0055-0060, wherein a threshold size of an object in the foreground is determined by the computing device, and a threshold filter is disclosed to segment an object of interest in the foreground and identify it from the background); identify depth information of the object of interest and depth information of a plurality of background objects included in an area excluding the object of interest in the first area (paras. 0037-0038, wherein depth information is captured by a depth camera alongside RGB images); identify a background object where the object of interest is located, from among the plurality of background objects, based on a difference between the depth information of the object of interest and the depth information of each of the plurality of background objects (para. 0054, 0066-0069, 0076-0081, and 0090-0093, wherein background objects are detected by depth differences using a geodesic distance metric; and wherein the object of interest is segmented out of the background images based on the geodesic distance as the background is eliminated), and wherein the processor, when executing the instructions, is further configured to identify a smallest value from among differences between the depth information of the object of interest and the depth information of each of the plurality of background objects (paras. 0066-0069, 0076-0081, and 0090-0093, wherein background objects are detected by depth differences using a geodesic distance metric; and wherein the object of interest is segmented out of the background images based on the geodesic distance as the background is eliminated); and identify a background object corresponding to the smallest value as a background object where the object of interest is located (paras. 0076-0081 and 0096-0101, wherein the depth information differences are geodesic distances calculated between different foreground and background objects within the image frame, and wherein segmented depth images can be used to eliminate background objects except for the background object closest to the object of interest).
Claim 10 is rejected, mutatis mutandis, for reasons similar to claim 1.
Regarding claim 6, Criminisi discloses all limitations of claim 1. Criminisi further discloses wherein the camera comprises a red-green-blue (RGB) photographing module and a depth photographing module (paras. 0043-0045, where the disclosed embodiment states that the RGB image and depth image can be used in tandem, wherein the RGB images are used to augment the depth images); and wherein the processor, when executing the instructions, is further configured to identify the first area of the threshold size including the object of interest in an RGB image obtained by the RGB photographing module (paras. 0034, 0043-0045, and 0055-0060, wherein a threshold size of an object in the foreground is determined by the computing device, and a threshold filter is disclosed to segment an object of interest in the foreground and identify it from the background); and identify depth information of the object of interest and depth information of the plurality of background objects included in an area excluding the object of interest in the first area based on a depth image corresponding to the RGB image obtained by the depth photographing module (paras. 0037-0038, wherein depth information is captured by a depth camera alongside RGB images).
Regarding claim 15, Criminisi discloses all limitations of claim 10. Criminisi further discloses wherein identifying the first area comprises: identifying the first area of the threshold size including the object of interest in red- green-blue (RGB) image obtained by the camera (paras. 0034, 0043-0045, and 0055-0060, wherein a threshold size of an object in the foreground is determined by the computing device, and a threshold filter is disclosed to segment an object of interest in the foreground and identify it from the background), and wherein the identifying depth information comprises identifying depth information of the object of interest and depth information of a plurality of background objects included in an area excluding the object of interest in the first area based on a depth image corresponding to the RGB image obtained by the camera (paras. 0037-0038, wherein depth information is captured by a depth camera alongside RGB images).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2, 9, 11, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Criminisi in view of Myers et al. (US PG Pub 20130182905, hereinafter “Myers”).
Regarding claims 2 and 11, Criminisi discloses all limitations of claims 1 and 10 respectively. Specifically, Criminisi discloses a segmentation method to detect a user from a background as a motion-based control mechanism for playing video games.
Criminisi does not disclose wherein the processor, when executing the instructions, is further configured to perform the method, the method comprising identifying an imaging angle of the camera with respect to the object of interest based on location information of the first area in the image; and identifying the background object where the object of interest is located, from among the plurality of background objects, based on height information of the camera, the imaging angle of the camera and the depth information of each of the plurality of background objects.
However, Myers discloses wherein the processor, when executing the instructions, is further configured to perform the method comprising identifying an imaging angle of the camera with respect to the object of interest based on location information of the first area in the image (paras. 0046 and 0055, wherein the camera data and metadata, such as depth and height, are collected from the images in the video stream, and wherein the camera height and tilt angle can be obtained from direct measurement or calibration); and identifying the background object where the object of interest is located, from among the plurality of background objects, based on height information of the camera, the imaging angle of the camera and the depth information of each of the plurality of background objects (paras. 0046, 0055, 0060-0062, and 0064-0065, wherein the height information and imaging angle of the camera are calculated from a combination of collected image stream data, metadata, and camera parameters obtained through direct measurement and calibration, and depth information is calculated using a depth sensor/RGBD sensor for background objects).
Specifically, Myers discloses a method for scene monitoring, wherein camera and depth information are used for scene occupancy detection (segmenting foreground people or objects of interest from the background).
Therefore, both Criminisi and Myers disclose foreground object of interest segmentation methods utilizing both RGB and depth images, wherein objects are segmented from backgrounds using depth distance information. Thus, it would have been obvious for one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the camera height and angle information determination of Myers within the apparatus and method of Criminisi as the application of a known technique to a known device in the same field of endeavor ready for improvement, yielding the predictable improvement of a more accurate depth distance calculation (taking height and angle into account) and, as a result, a more accurate segmentation of an object of interest.
Regarding claims 9 and 18, Criminisi discloses all limitations of claims 1 and 10, respectively. Criminisi further discloses a memory configured to store map information (paras. 0045 and 0076, wherein para. 0045 discloses the memory and para. 0076 discloses the depth values of image elements being reconstructed as 3D depth map) and wherein the processor, when executing the instructions, is further configured to perform a method, the method further comprising storing map information (para. 0076 for storing information obtained from the depth sensor as a 3D depth map).
Criminisi does not disclose identifying location information of the object of interest based on location information of the identified background object, or updating the map information based on the identified location information of the object of interest.
However, Myers discloses identifying location information of the object of interest based on location information of the identified background object (paras. 0084-0086, wherein the identified background object can be identified as an object occluding the objects of interest) and wherein the map information is updated based on the identified location information of the object of interest (para. 0069 for the depth map creation and paras. 0085-0086 for updating based on locating object(s) of interest within different image frames).
Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the disclosure of Myers within the apparatus and method of Criminisi according to the rationale of claim 2.
Claims 3, 5, 7, 12, 14, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Criminisi in view of Hillborg (US PG Pub 20190311493).
Regarding claims 3 and 12, Criminisi discloses all limitations of claim 1. Criminisi further discloses identifying depth information of a plurality of background objects included in an area (paras. 0054, 0066-0069, 0076-0081, and 0090-0093, wherein background objects are detected by depth differences using a geodesic distance metric; wherein the object of interest is segmented out of the background images based on the geodesic distance as the background is eliminated); identifying the background object where the object of interest is located, from among the plurality of background objects, based on depth information of the object of interest identified in the first area, depth information of the plurality of background objects identified in the first area and depth information of the plurality of background objects identified in the second area (paras. 0054, 0066-0069, 0076-0081, and 0090-0093, wherein background objects are detected by depth differences using a geodesic distance metric; wherein the object of interest is segmented out of the background images based on the geodesic distance as the background is eliminated; and wherein one having ordinary skill in the art would reasonably be able to apply this method of object of interest and background image identification in different image areas).
Criminisi does not disclose identifying a second area corresponding to the first area in the subsequent image.
However, Hillborg discloses identifying a second area corresponding to the first area in the subsequent image (para. 0050, wherein the second area contains the analogous object of interest, and wherein the second area falls within a single standard of deviation of the initial area with respect to the image frame).
Specifically, Hillborg discloses a method and system for identifying image objects using neural networks. Therefore, both Criminisi and Hillborg both disclose methods and systems of object-of-interest identification utilizing a form of machine learning framework to segment regions of interest.
Thus, it would have been obvious for one having ordinary skill in the art prior to the effective filing date of the claimed invention to have implemented the second area identification of Hillborg within the method and system of Criminisi as the application of the known method of Hillborg to the known system of Criminisi to yield the predictable result of an object identification method with a wider range of image areas for faster, easier, and multi-angled identification of image objects of interest within a plurality of background objects.
Regarding claims 5 and 14, Criminisi discloses all limitations of claim 1. Criminisi further discloses wherein the processor, when executing the instructions, is further configured to execute a method, the method further comprising obtaining the first area of the threshold size including the object of interest by inputting the obtained image to an image segmentation model (paras. 0034, 0055-0060, 0096-0104, and 0130-0132, wherein a threshold size of an object in the foreground is determined by the computing device, a threshold filter is disclosed to segment an object of interest in the foreground and identify it from the background, and a random forest classifier for classifying image objects is employed to segment the image by foreground/object of interest and background), and wherein the image segmentation model is trained to, based on the image being input, output the object of interest included in the image and area identification information including a plurality of background objects (paras. 0130-0133, wherein the ID information is whether an object is in the foreground or the background, and wherein the object of interest is segmented out of the background image).
Criminisi does not disclose wherein the image segmentation model is a neural network.
However, Hillborg discloses wherein a particular image segmentation algorithm for extracting specific image crops/segments containing objects of interest is a neural network (para. 0039-0042, specifically where the neural networks extract regions of interest). Specifically, Hillborg discloses a method and system for identifying image objects using neural networks. Therefore, both Criminisi and Hillborg both disclose methods and systems of object-of-interest identification utilizing a form of machine learning framework to segment regions of interest. Thus, it would have been obvious for one having ordinary skill in the art prior to the effective filing date of the claimed invention to have substituted the random forest classifier of Criminisi with the neural network of Hillborg as a simple substitution known to those having ordinary skill in the art.
Regarding claims 7 and 16, Criminisi discloses all limitations of claim 1. Criminisi further discloses wherein the processor, when executing the instructions, is further configured to execute a method, the method further comprising identifying depth information of the plurality of background objects based on depth information of each segmentation area (para. 0054, 0066-0069, 0076-0081, and 0090-0093, wherein background objects are detected by depth differences using a geodesic distance metric; and wherein the object of interest is segmented out of the background images based on the geodesic distance as the background is eliminated).
Criminisi does not disclose wherein the processor-executed method is further configured to obtain a segmentation area corresponding to each of the plurality of background objects by inputting the first area to a neural network model, wherein the neural network model is trained to, based on an image being input, output area identification information corresponding to each of the plurality of background objects included in the image
However, Hillborg discloses wherein the processor-executed method is further configured to obtain a segmentation area corresponding to each of the plurality of background objects by inputting the first area to a neural network model (paras. 0046-0050, and 0054-0058, wherein multiple image segmentation areas are input into a neural network model to identify pre-specified features, such as background images, and bounding boxes resulting from measurements of different metrics (in this case, the combination with Criminisi would enable comparison of depth metrics) in order to identify background objects), and wherein the neural network model is trained to, based on an image being input, output area identification information corresponding to each of the plurality of background objects included in the image (paras. 0130-0133, wherein the ID information is whether an object is in the foreground or the background, and wherein the object of interest is segmented out of the background image).
Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the disclosures according to the method of claims 5 and 14.
Claims 4 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Criminisi in view of Wang et al. (Chinese PG Pub 110136174, hereinafter “Wang”).
Regarding claims 4 and 13, Criminisi discloses all limitations of claim 1. Criminisi further discloses identifying depth information of the object of interest and depth information of a plurality of background objects included in an area excluding the object of interest in an area (para. 0054, 0066-0069, 0076-0081, and 0090-0093, wherein background objects are detected by depth differences using a geodesic distance metric; and wherein the object of interest is segmented out of the background images based on the geodesic distance as the background is eliminated).
Criminisi does not disclose wherein the processor, when executing the instructions, is further configured to
based on identifying the first area, identify whether a ratio of a size of the object of interest in the first area to a size of the first area is equal to or greater than a threshold ratio; or
based on identifying that the ratio is equal to or greater than the threshold ratio, identify a third area larger than the threshold size in the image.
However, Wang discloses wherein a processor, when executing the instructions, is further configured to:
based on identifying the first area, identify whether a ratio of a size of the object of interest in the first area to a size of the first area is equal to or greater than a threshold ratio (paras. 0015 and 0036, detailing that the ratio of the object sizes of objects of interest within frames in a series of frames are compared against a threshold value to determine whether they belong to the same object of interest to be tracked); and
based on identifying that the ratio is equal to or greater than the threshold ratio, identify a third area larger than the threshold size in the image (para. 0133, directly following the comparison of the thresholds of intermediate portions of an ROI which exceed a threshold, these intermediate regions of interest may be combined into a larger region of interest).
Specifically, Wang discloses a three-dimensional target tracking system wherein the size ratios of foreground objects are compared with a threshold value to identify whether they correspond to the same object within motion. Therefore, Criminisi and Wang both disclose methods and systems of three-dimensional object of interest tracking using depth information, changes between foreground object motion relative to background objects, and sizes of image objects to determine movement based on thresholded ratios. Thus, it would have been obvious for one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the size ratio and thresholding method of Wang within the method and system of Criminisi as the application of a known technique to a known device ready for improvement to yield the predictable result of an object tracking system robust to objects of interest being located at the edge of a frame or the foreground region.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROHAN TEJAS MUKUNDHAN whose telephone number is (571)272-2368. The examiner can normally be reached Monday - Friday 9AM - 6PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at 5712723838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROHAN TEJAS MUKUNDHAN/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698