Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 18 May 2026 has been entered.
Response to Amendment
In response to the amendment to independent claim 1, the previously-delivered objection is withdrawn.
In response to the amendment to independent claims 1 and 10, the rejection of independent claims 1 and 10 under 35 U.S.C. § 102 as being anticipated by Criminisi et al. are withdrawn. However, upon further consideration, a new ground of rejection is made under 35 U.S.C. 103 as being unpatentable over the aforementioned Criminisi reference in view of Myers et al. (full citations within PTO-892 form)
Response to Arguments
In response to Applicant’s arguments with respect to the IDS, specifically the acknowledgement of the IDS’s citation in a counterpart foreign application under 37 CFR 1.97(e)(1), the IDS of 20 May 2026 has been considered, and an annotated version is included within this Office Action.
In response to applicant’s arguments that Criminisi fails to disclose a variety of limitations of both previously presented and the newly-amended claim 1, Examiner notes that a recitation of the intended use of the claimed invention must result in a structural difference between the claimed invention and the prior art in order to patentably distinguish the claimed invention from the prior art. If the prior art structure is capable of performing the intended use, then it meets the claim. Thus, even if the disclosed geodesic distance metric is employed within the disclosure of Criminisi to calculated differences to adjudge foreground and background differences for background removal, it is still more than capable of identifying objects of interesting, thresholding, and calculating object depth while obtaining map information based on location information of the object of interest as recited within the independent claims of the instant application. Thus, Examiner maintains that Criminisi discloses all limitations of the broadest reasonable interpretation of previously presented claim 1. The rejection under 35 U.S.C.§ 102 is withdrawn, however, as newly-included limitations are not disclosed by Criminisi, but a new ground of rejection is applied under 35 U.S.C. § 103 as being unpatentable over Criminisi in view of Myers.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 9-11, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Criminisi et al. (US PG Pub 20140126821, hereinafter “Criminisi”) in view of Myers et al. (US PG Pub 20130182905, hereinafter “Myers”).
Regarding claim 1, Criminisi discloses an electronic apparatus comprising:
a camera comprising a red-green-blue (RGB) photographing module and a depth photographing module (paras. 0043-0045, where the disclosed embodiment states that the RGB image and depth image can be used in tandem, wherein the RGB images are used to augment the depth images);
memory storing instructions (paras. 0045 and 0076, wherein para. 0045 discloses the memory and para. 0076 discloses the depth values of image elements being reconstructed as 3D depth map);
and a processor (para. 0044, where the processor is configured to be in communication with the image sensor), wherein the instructions, when executed by the processor, cause the electronic apparatus to:
identify a first area of a threshold size in an image obtained by the camera, the first area including an object of interest among a plurality of objects identified from the obtained image (paras. 0034 and 0055-0060, wherein a threshold size of an object in the foreground is determined by the computing device, and a threshold filter is disclosed to segment an object of interest in the foreground and identify it from the background);
identify, from the obtained image, depth information of the object of interest and depth information of a plurality of background objects, among the plurality of objects identified from the obtained image, included in an area excluding the object of interest in the first area, wherein the depth information of the plurality of background objects includes distances from the camera to each of the plurality of background objects (paras. 0034, 0043-0045, and 0055-0060, wherein a threshold size of an object in the foreground is determined by the computing device, and a threshold filter is disclosed to segment an object of interest in the foreground and identify it from the background; and paras. 0037-0038, wherein depth information is captured by a depth camera alongside RGB images for the plurality of foreground and background objects);
and identify, from the obtained image, a background object where the object of interest is located, from among the plurality of background objects, based on a difference between the depth information of the object of interest and the depth information of each of the plurality of background objects (para. 0054, 0066-0069, 0076-0081, and 0090-0093, wherein background objects are detected by depth differences using a geodesic distance metric; and wherein the object of interest is segmented out of the background images based on the geodesic distance as the background is eliminated),
and wherein the instructions, when executed by the processor, further cause the electronic apparatus to:
identify, from the obtained image, a smallest value from among differences between the depth information of the object of interest and the depth information of each of the plurality of background objects identified from the obtained image (paras. 0066-0069, 0076-0081, and 0090-0093, wherein background objects are detected by depth differences using a geodesic distance metric; and wherein the object of interest is segmented out of the background images based on the geodesic distance as the background is eliminated);
and identify a background object corresponding to the smallest value as a background object where the object of interest is located (paras. 0076-0081 and 0096-0101, wherein the depth information differences are geodesic distances calculated between different foreground and background objects within the image frame, and wherein segmented depth images can be used to eliminate background objects except for the background object closest to the object of interest),
and wherein the instructions, when executed by the processor, further cause the electronic apparatus to identify the first area of the threshold size including the object of interest in an RGB image obtained by the RGB photographing module (paras. 0034, 0043-0045, and 0055-0060, wherein a threshold size of an object in the foreground is determined by the computing device, and a threshold filter is disclosed to segment an object of interest in the foreground and identify it from the background); and identify depth information of the object of interest and depth information of the plurality of background objects included in the area excluding the object of interest in the first area based on a depth image corresponding to the RGB image obtained by the depth photographing module (paras. 0037-0038, wherein depth information is captured by a depth camera alongside RGB images).
Specifically, Criminisi discloses a segmentation method to detect a user from a background as a motion-based control mechanism for playing video games.
Criminisi does not disclose wherein the instructions, when executed by the processor, further cause the electronic apparatus to:
identify location information of the object of interest based on location information of the identified background object; and obtain map information based on the identified location information of the object of interest; or
wherein the smallest value is a background object on top of which the object of interest is located.
However, Myers discloses wherein the instructions, when executed by the processor, further cause the electronic apparatus to:
identify location information of the object of interest based on location information of the identified background object (paras. 0084-0086, wherein the identified background object can be identified as an object occluding the objects of interest); and
obtain map information based on the identified location information of the object of interest (para. 0069 for the depth map creation and paras. 0085-0086 for updating based on locating object(s) of interest within different image frames); and
wherein the smallest value is a background object on top of which the object of interest is located (para. 0099, “ Another example of a general application of the above embodiments is to perform object tracking to determine when a person falls down. For example, a captured image may have the shape and size of a person, but the height information (that may be obtained from depth information) may show that the person's head is near to the ground (e.g., one foot off the ground), may indicate that a person has fallen down or is lying down…such, a top of someone's head may have a different shade or color from a point lower on the person's head.”, wherein Myers specifically discloses detection of a body based on planes and locations in frames; wherein the interactions and interfaces of planes of objects and tracking objects would be recognized by an ordinarily skilled artisan as being in contact with a surface, and wherein the minimal depth difference would be known to that ordinarily skilled artisan).
Specifically, Myers discloses a method for scene monitoring, wherein camera and depth information are used for scene occupancy detection (segmenting foreground people or objects of interest from the background).
Therefore, both Criminisi and Myers disclose foreground object of interest segmentation methods utilizing both RGB and depth images, wherein objects are segmented from backgrounds using depth distance information. Thus, it would have been obvious for one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the camera height and angle information determination of Myers within the apparatus and method of Criminisi as the application of a known technique to a known device in the same field of endeavor ready for improvement, yielding the predictable improvement of a more accurate depth calculation and map updating which would result in a more accurate segmentation of an object of interest.
Claim 10 is rejected, mutatis mutandis, for reasons similar to claim 1.
Regarding claims 2 and 11, Criminisi discloses all limitations of claims 1 and 10 respectively. Criminisi does not disclose wherein the instructions, when executed by the processor, further causes the electronic apparatus to identify an imaging angle of the camera with respect to the object of interest based on location information of the first area in the image; and identifying the background object where the object of interest is located, from among the plurality of background objects, based on height information of the camera, the imaging angle of the camera and the depth information of each of the plurality of background objects.
However, Myers discloses wherein the instructions, when executed by the processor, further causes the electronic apparatus to identify an imaging angle of the camera with respect to the object of interest based on location information of the first area in the image (paras. 0046 and 0055, wherein the camera data and metadata, such as depth and height, are collected from the images in the video stream, and wherein the camera height and tilt angle can be obtained from direct measurement or calibration); and identifying the background object where the object of interest is located, from among the plurality of background objects, based on height information of the camera, the imaging angle of the camera and the depth information of each of the plurality of background objects (paras. 0046, 0055, 0060-0062, and 0064-0065, wherein the height information and imaging angle of the camera are calculated from a combination of collected image stream data, metadata, and camera parameters obtained through direct measurement and calibration, and depth information is calculated using a depth sensor/RGBD sensor for background objects).
Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the disclosure of Myers within the apparatus and method of Criminisi according to the rationale of claim 1.
Regarding claims 9 and 18, Criminisi discloses all limitations of claims 1 and 10, respectively. Criminisi further discloses a memory configured to store map information (paras. 0045 and 0076, wherein para. 0045 discloses the memory and para. 0076 discloses the depth values of image elements being reconstructed as 3D depth map) and wherein the instructions, when executed by the processor, further cause the electronic apparatus toto perform a method, the method further comprising storing map information (para. 0076 for storing information obtained from the depth sensor as a 3D depth map).
Criminisi does not disclose identifying location information of the object of interest based on location information of the identified background object, or updating the map information based on the identified location information of the object of interest.
However, Myers discloses identifying location information of the object of interest based on location information of the identified background object (paras. 0084-0086, wherein the identified background object can be identified as an object occluding the objects of interest) and wherein the map information is updated based on the identified location information of the object of interest (para. 0069 for the depth map creation and paras. 0085-0086 for updating based on locating object(s) of interest within different image frames).
Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the disclosure of Myers within the apparatus and method of Criminisi according to the rationale of claim 1.
Claims 3, 5, 7, 12, 14, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Criminisi in view of Myers and in further view of Hillborg (US PG Pub 20190311493).
Regarding claims 3 and 12, Criminisi in view of Myers discloses all limitations of claim 1. Criminisi further discloses identifying depth information of a plurality of background objects included in an area (paras. 0054, 0066-0069, 0076-0081, and 0090-0093, wherein background objects are detected by depth differences using a geodesic distance metric; wherein the object of interest is segmented out of the background images based on the geodesic distance as the background is eliminated); identifying the background object where the object of interest is located, from among the plurality of background objects, based on depth information of the object of interest identified in the first area, depth information of the plurality of background objects identified in the first area and depth information of the plurality of background objects identified in the second area (paras. 0054, 0066-0069, 0076-0081, and 0090-0093, wherein background objects are detected by depth differences using a geodesic distance metric; wherein the object of interest is segmented out of the background images based on the geodesic distance as the background is eliminated; and wherein one having ordinary skill in the art would reasonably be able to apply this method of object of interest and background image identification in different image areas).
Criminisi in view of Myers does not disclose identifying a second area corresponding to the first area in the subsequent image.
However, Hillborg discloses identifying a second area corresponding to the first area in the subsequent image (para. 0050, wherein the second area contains the analogous object of interest, and wherein the second area falls within a single standard of deviation of the initial area with respect to the image frame).
Specifically, Hillborg discloses a method and system for identifying image objects using neural networks. Therefore, both Criminisi in view of Myers and Hillborg both disclose methods and systems of object-of-interest identification utilizing a form of machine learning framework to segment regions of interest.
Thus, it would have been obvious for one having ordinary skill in the art prior to the effective filing date of the claimed invention to have implemented the second area identification of Hillborg within the method and system of Criminisi in view of Myers as the application of the known method of Hillborg to the known system of Criminisi in view of Myers to yield the predictable result of an object identification method with a wider range of image areas for faster, easier, and multi-angled identification of image objects of interest within a plurality of background objects.
Regarding claims 5 and 14, Criminisi in view of Myers discloses all limitations of claim 1. Criminisi further discloses wherein the processor, when executing the instructions, is further configured to execute a method, the method further comprising obtaining the first area of the threshold size including the object of interest by inputting the obtained image to an image segmentation model (paras. 0034, 0055-0060, 0096-0104, and 0130-0132, wherein a threshold size of an object in the foreground is determined by the computing device, a threshold filter is disclosed to segment an object of interest in the foreground and identify it from the background, and a random forest classifier for classifying image objects is employed to segment the image by foreground/object of interest and background), and wherein the image segmentation model is trained to, based on the image being input, output the object of interest included in the image and area identification information including a plurality of background objects (paras. 0130-0133, wherein the ID information is whether an object is in the foreground or the background, and wherein the object of interest is segmented out of the background image).
Criminisi in view of Myers does not disclose wherein the image segmentation model is a neural network.
However, Hillborg discloses wherein a particular image segmentation algorithm for extracting specific image crops/segments containing objects of interest is a neural network (para. 0039-0042, specifically where the neural networks extract regions of interest). Specifically, Hillborg discloses a method and system for identifying image objects using neural networks. Therefore, both Criminisi in view of Myers and Hillborg both disclose methods and systems of object-of-interest identification utilizing a form of machine learning framework to segment regions of interest. Thus, it would have been obvious for one having ordinary skill in the art prior to the effective filing date of the claimed invention to have substituted the random forest classifier of Criminisi in view of Myers with the neural network of Hillborg as a simple substitution known to those having ordinary skill in the art.
Regarding claims 7 and 16, Criminisi in view of Myers discloses all limitations of claim 1. Criminisi further discloses wherein the processor, when executing the instructions, is further configured to execute a method, the method further comprising identifying depth information of the plurality of background objects based on depth information of each segmentation area (para. 0054, 0066-0069, 0076-0081, and 0090-0093, wherein background objects are detected by depth differences using a geodesic distance metric; and wherein the object of interest is segmented out of the background images based on the geodesic distance as the background is eliminated).
Criminisi in view of Myers does not disclose wherein the processor-executed method is further configured to obtain a segmentation area corresponding to each of the plurality of background objects by inputting the first area to a neural network model, wherein the neural network model is trained to, based on an image being input, output area identification information corresponding to each of the plurality of background objects included in the image
However, Hillborg discloses wherein the processor-executed method is further configured to obtain a segmentation area corresponding to each of the plurality of background objects by inputting the first area to a neural network model (paras. 0046-0050, and 0054-0058, wherein multiple image segmentation areas are input into a neural network model to identify pre-specified features, such as background images, and bounding boxes resulting from measurements of different metrics (in this case, the combination with Criminisi in view of Myers would enable comparison of depth metrics) in order to identify background objects), and wherein the neural network model is trained to, based on an image being input, output area identification information corresponding to each of the plurality of background objects included in the image (paras. 0130-0133, wherein the ID information is whether an object is in the foreground or the background, and wherein the object of interest is segmented out of the background image).
Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the disclosures according to the method of claims 5 and 14.
Claims 4 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Criminisi in view of Myers and in further view of Wang et al. (Chinese PG Pub 110136174, hereinafter “Wang”).
Regarding claims 4 and 13, Criminisi in view of Myers discloses all limitations of claim 1. Criminisi further discloses identifying depth information of the object of interest and depth information of a plurality of background objects included in an area excluding the object of interest in an area (para. 0054, 0066-0069, 0076-0081, and 0090-0093, wherein background objects are detected by depth differences using a geodesic distance metric; and wherein the object of interest is segmented out of the background images based on the geodesic distance as the background is eliminated).
Criminisi in view of Myers does not disclose wherein the instructions, when executed by the processor, is further configured to
based on identifying the first area, identify whether a ratio of a size of the object of interest in the first area to a size of the first area is equal to or greater than a threshold ratio; or
based on identifying that the ratio is equal to or greater than the threshold ratio, identify a third area larger than the threshold size in the image.
However, Wang discloses wherein a processor, when executing the instructions, is further configured to:
based on identifying the first area, identify whether a ratio of a size of the object of interest in the first area to a size of the first area is equal to or greater than a threshold ratio (paras. 0015 and 0036, detailing that the ratio of the object sizes of objects of interest within frames in a series of frames are compared against a threshold value to determine whether they belong to the same object of interest to be tracked); and
based on identifying that the ratio is equal to or greater than the threshold ratio, identify a third area larger than the threshold size in the image (para. 0133, directly following the comparison of the thresholds of intermediate portions of an ROI which exceed a threshold, these intermediate regions of interest may be combined into a larger region of interest).
Specifically, Wang discloses a three-dimensional target tracking system wherein the size ratios of foreground objects are compared with a threshold value to identify whether they correspond to the same object within motion. Therefore, Criminisi in view of Myers and Wang both disclose methods and systems of three-dimensional object of interest tracking using depth information, changes between foreground object motion relative to background objects, and sizes of image objects to determine movement based on thresholded ratios. Thus, it would have been obvious for one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the size ratio and thresholding method of Wang within the method and system of Criminisi in view of Myers as the application of a known technique to a known device ready for improvement to yield the predictable result of an object tracking system robust to objects of interest being located at the edge of a frame or the foreground region.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GREGORY A MORSE whose telephone number is (571)272-3838. The examiner can normally be reached M-F 7:30-4.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights