DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because of the following informalities:
The disclosure is objected to because it contains embedded hyperlinks and/or other forms of browser-executable code. Applicant is required to delete the embedded hyperlinks and/or other forms of browser-executable code; references to websites should be limited to the top-level domain name without any prefix such as http:// or other browser-executable code. See MPEP § 608.01.
There are hyperlinks in paragraphs 00063, 00064, 00065, 00066, and 000370.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 14 and 18 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 14 recites the limitation “the output image” in line 3. There is insufficient antecedent basis for this limitation in the claim.
Claim 18 recites the limitation “the neural network” in line 3. There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 16 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. (U.S. Patent No. 10,068,336, hereafter referred to as Zhang) in view of Popp et al. (NPL "A novel guidance and navigation system for MAVs capable of autonomous collision-free entering of buildings," 2015, hereafter referred to as Popp).
Regarding Claim 1, Zhang teaches a method for real-time detection and tracking of potential passages in an environment (Claims 7 & 8, Zhang teaches a computer implemented method for doorway detection, in which line segments are extracted from an image frame captured in an indoor environment, determining the existence of a vanishing point, then performing either side doorway detection via a side doorway detection module or frontal doorway detection via a frontal doorway detection module and outputting a description of the detected doorways. Each doorway detection module has a corresponding edge tracker for tracking the vertical edges of a door.), the method comprising: a) detecting one or more passages in one or more frames of image data (Col. 10, lines 20-22, Zhang teaches using edge-based door detection to detect doors based on the edges in each frame of the video input.); c) tracking one or more points between frames of image data for each of the one or more detected passages in one or more frames of image data (Col. 13, lines 12-17, 39-40, Zhang teaches first sampling a set of points along the line (edge) in a previous frame. The tracking algorithm then searches for a matching point for each of the sampled points in the line in the current frame. This matching process is done for each sampled point.); d) assigning one or more passages detected in a frame of image data to one or more previously detected passages in a different frame of image data based on i) one or more edges of the one or more detected passages (Col. 12, line 66-67, Col. 13, lines 1-16, Zhang teaches a line (edge) tracking process in the door detection system. When a door is detected, its two near-vertical lines initialize a line tracker. The tracker will keep tracking these lines in following image frames until it loses tracking. Assuming a door line is detected in the previous frame, the line tracker searches for the new location of this line in the new frame.).; and iii) the tracking one or more points between frames of image data for each of the one or more detected passages in the one or more frames of image data (Fig. 6, Col. 13, lines 7-17, Zhang teaches point searching and matching for line tracking in two consecutive frames. A set of points (i.e., a sampled point, Pit) along the line is first sampled uniformly in the previous frame (Ft). The tracking algorithm then searches for a matching point (Pit+1) for each of the sampled points in the line in the current frame Ft+1.).
Popp is in the same field of art of detection and tracking of passages in an environment. Further, Popp teaches b) extracting one or more corners for each of the one or more detected passages (Fig. 3, “Target Detection, Tracking and Localisation” Section, Popp discloses a camera that serves to detect the striking corners of the target (denoted C1-C4 in Fig. 3)); and (d) assigning one or more passages detected in a frame of image data to one or more previously detected passages in a different frame of image data based on ii) the one or more corners (“Tracking” Section, Popp discloses tracking the detected window corners in the camera image using an optical flow algorithm. After each tracking step, an integrity check of the tracked features is performed. The ratio of the target’s width and the target’s height are checked against the previously detected target and it is checked whether the edges of the target are still orthogonal to each other or not. The examiner interprets that orthogonal, which is defined as “intersecting or lying at right angles” is synonymous to checking whether the edges still form a corner, as they did in previous frames.).
Therefore it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Zhang by detecting one or more corners for each detected passage and tracking the detected corners between image frames that is taught by Popp, to make the invention that detects passages such as doorways, windows, etc. between frames using both edge and corner information; thus one of ordinary skill in the art would have been motivated to combine the references to enable detection of both frontal and side doorways since for frontal doorway detection, the method would require at least two detected corners, whereas for this reason, such a method would not work for detecting side doorways when either the top or bottom frames are not in view or the corners are not detected (Zhang, Col. 1, lines 48-62).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 16, Zhang in view of Popp discloses the method of claim 1, wherein the tracking one or more points between frames of image data for each of the one or more detected passages in one or more frames of image data comprises: applying at least one of intersection over union, optical flow, appearance descriptor, or the DeepSort algorithm to the one or more detected passages (“Tracking” Section, Popp discloses tracking the detected window corners in the camera image using a sparse iterative version of the Lucas-Kanade Optical Flow algorithm. Under Broadest Reasonable Interpretation, the Examiner interprets the claim language of “at least one of” to be only one limitation has to be met.).
In regards to Claim 20, Zhang discloses a system for real-time detection and tracking of potential passages in an environment (Claims 1 & 2, Zhang teaches a system for doorway detection, wherein the system comprises one or more processors and a non-transitory computer-readable medium having executable instructions encoded thereon such that when executed, the one or more processors perform the operation of extracting salient line segments from an image frame captured of an indoor environment, determining the presence of a vanishing point in the image frame, if a vanishing point is detected with a confidence score meeting or exceeding the predetermined confidence score, then performing side doorway detection via a side doorway detection module. If the vanishing point is detected with a confidence score below the predetermined confidence score, then performing frontal doorway detection via a frontal doorway detection module.), the system comprising: circuitry for detecting one or more passages in one or more frames of image data (Col. 7, lines 26-50, Col. 9, lines 51-62, Col. 6, lines 4-52, Fig. 1, Zhang teaches computer executable instructions, such as program modules, which include routines, programs, objects, components, and/or data structures that are configured to perform particular tasks and are executed by a computer system. The computer system may include an address/data bus, one or more processing units, and one or more data storage units. The computer system consists of several modules including a side corridor detection module and a frontal doorway detection module.); circuitry for tracking one or more points between frames of image data for each of the one or more detected passages in one or more frames of image data (Col. 9, lines 51-62, Fig. 3, reference characters 310 and 312, Col. 13, lines 14-17, Zhang teaches a line tracking module executed by the computer system. The tracking algorithm searches for a matching point in the previous frame for each of the sampled points in the line of the current frame.); and circuitry for assigning one or more passages detected in a frame of image data to one or more previously-detected passages in a different frame of image data based on one or more edges of the one or more detected passages (Col. 9, lines 51-62, Fig. 3, reference characters 310 and 312, Col. 13, lines 7-9, Zhang teaches a line tracking module executed by the computer system. Assuming a door line is detected in a previous frame, the line tracking module searches for the new location of this line in the new frame.); and the circuitry for tracking one or more points between frames of image data for each of the one or more detected passages in the one or more frames of image data (Col. 10, lines 28-37, Col. 6, lines 13-15, Zhang teaches once a door has been detected, an edge tracker will be initiated to track the door’s vertical edges. Depending on the door detection module that is triggered (side or frontal door detection modules), the tracked lines from the edge tracker are used either as candidate door lines for door hypothesis generation or directly as tracked doors. The door detection modules are executed by the computer system.).
Zhang does not explicitly disclose circuitry for extracting one or more corners for each of the one or more detected passages and circuitry for assigning one or more passages detected in a frame of image data to one or more previously-detected passages in a different frame of image data based on the one or more corners.
Popp is in the same field of art of detection and tracking of passages in an environment. Further, Popp discloses circuitry for extracting one or more corners for each of the one or more detected passages (“Target, Detection, Tracking and Localisation” Section, “Hardware” Section, Popp teaches using sensor information of a camera and a laser rangefinder to detect and track the target during the whole flight. The camera serves to detect the striking corners of the target (C1-C4). An embedded computer is used to carry out the presented algorithms, including the detection and tracking algorithms.); and circuitry for assigning one or more passages detected in a frame of image data to one or more previously-detected passages in a different frame of image data based on the one or more corners (“Tracking” Section, “Target Detection, Tracking, and Localisation” Section, Popp discloses tracking the detected window corners in the camera image using an Optical Flow algorithm. After each tracking step, an integrity check of the tracked features is completed. This means that it is checked if the ratio of target width and target height is correct and it is checked whether the edges of the target’s contour are still orthogonal to each other or not. The target (passage) is detected throughout the whole flight using sensor information of a camera and a laser rangefinder. The camera detects the corners of the target. An embedded computer is used to carry out the presented algorithms, such as the detection and tracking algorithms.).
Therefore it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Zhang by adding the circuitry required to extract corners from the passages and for tracking the corners between frames that is taught by Popp, to make the invention that detects passages such as doorways and windows between image frames based on tracking both edge and corner information; thus one of ordinary skill in the art would have been motivated to combine the references to enable reliable detection of passages such as doors and windows because although corner detection alone may work for identifying front-facing doors and windows, it does not always work for detecting side doorways when either the top or bottom frames of the door or window are not observed, and therefore the corners cannot be detected in those situations (Zhang, Col. 1, lines 54-62). Combining edge and corner tracking creates a more robust solution that is able to handle more complex passage tracking situations.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim(s) 2 and 4 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. (U.S. Patent No. 10,068,336, hereafter referred to as Zhang) in view of Popp et al. (NPL "A novel guidance and navigation system for MAVs capable of autonomous collision-free entering of buildings," 2015, hereafter referred to as Popp) in further view of Dai et al. (NPL “Residential building façade segmentation in the urban environment,” 2021, hereafter referred to as Dai).
Regarding Claim 2, Zhang in view of Popp discloses the method of claim 1.
Zhang in view of Popp does not explicitly disclose wherein the detecting one or more passages in one or more frames of image data comprises: computing semantic segmentation of any passages in the one or more frames of image data; and computing an approximate bounding box for each detected passage in the one or more frames of image data.
Dai is in the same field of art of collecting key building geometry information, such as information about the location and size of walls, windows, and doors in the captured image frames. Further, Dai discloses wherein the detecting one or more passages in one or more frames of image data comprises (Section 1, Dai teaches collecting image data containing substantial building information which can be used to extract building geometry information. For example, the location of smaller façade components such as windows and doors.): computing semantic segmentation of any passages in the one or more frames of image data (Fig. 3, Section 2, Dai teaches two different semantic segmentation models. The first model segments the relatively smaller features of the building including windows, doors, and chimneys. The second model is designed for the features that take up a larger number of pixels such as the wall and roof.); and computing an approximate bounding box for each detected passage in the one or more frames of image data (Section 2.3, Fig. 3, Dai teaches automatically generating bounding box information by calculating the minimum bounding rectangles of the pixel-wise annotations to use the object detection model. For each annotation patch, its MBR coordinates are calculated. The pixel-wise annotations contain labels for all target objects appearing in an image, including windows, doors, etc. Figure 3 depicts the pixel-wise annotation samples.).
Therefore it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Zhang in view of Popp by computing semantic segmentation on the images to identify potential passages, such as windows and doors and computing bounding boxes for the detected passages that is taught by Dai, to make the invention that is able to distinguish passages such as doors and windows from other building features/classes such as roofs, walls and the background, as well as compute bounding boxes to mitigate the class imbalance problem; thus one of ordinary skill in the art would have been motivated to combine the references to categorize the different classes of objects within each image frame and distinguish objects from other types of objects and the background to more accurately identify passages (Dai, Sections 2.1 and 2.3).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 4, Zhang in view of Popp discloses the method of claim 1.
Zhang in view of Popp does not disclose wherein the detecting one or more passages in each frame of image data is carried out using at least one U-net convolutional neural network.
Dai is in the same field of art of collecting key building geometry information, such as information about the location and size of walls, windows, and doors in the captured image frames. Further, Dai teaches wherein the detecting one or more passages in each frame of image data is carried out using at least one U-net convolutional neural network (Section 2, Fig. 5, Dai teaches two different semantic segmentation models derived from the U-net architecture.).
Therefore it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Zhang in view of Popp by using a U-net architecture to detect passages, that is taught by Dai, to make the invention that allows for features representing small object information, such as windows, doors, and chimneys to be transmitted to higher levels of the network, thereby helping to mitigate the class imbalance problem; thus one of ordinary skill in the art would have been motivated to combine the references to retain small object (passage) information more accurately. Since the images contain a number of small objects, such as windows and doors, the benefits of using the symmetric U-net are highly relevant to solving the problem (Dai, Section 2.2).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. (U.S. Patent No. 10,068,336, hereafter referred to as Zhang) in view of Popp et al. (NPL "A novel guidance and navigation system for MAVs capable of autonomous collision-free entering of buildings," 2015, hereafter referred to as Popp) in further view of Fahimi et al. (NPL "A Vision-Based Guidance Algorithm for Entering Buildings Through Windows for Delivery Drones", hereafter referred to as Fahimi).
Regarding Claim 14, Zhang in view of Popp disclose the method of claim 1.
Zhang in view of Popp does not disclose wherein detecting one or more passages in one or more frames of image data further comprises: applying a threshold to each layer of the output image with a probability of 95% of being a member of a semantic class.
Fahimi is in the same field of detecting boundaries of a passage, specifically, a window portal section. Further, Fahimi discloses wherein detecting one or more passages in one or more frames of image data further comprises (Abstract, Fig. 11, Fahimi teaches an image-based optimization scheme to detect the exact boundaries of a window portal section. Figure 11 shows accurate portal predictions by the DAE method.): applying a threshold to each layer of the output image with a probability of 95% of being a member of a semantic class (“Finding the Global Optimum” Section, “Dark Area Extraction” Section, Fig. 7, Fahimi teaches first dividing the image into a number of segments, each representing a distinct section of the building façade (a window glass, portal section, window frame, etc.). Second, defining a cost function that is minimized when the correct entrance section is completely white and all of the other sections are black. To achieve this, a dynamic thresholding technique is used, in which the desired threshold parameter leads to a binary image containing a solid white polygon for the window portal section with no junctions with any other white blob. Figure 4 depicts samples of the binary image corresponding to a set of different threshold values (τ). Figures 7(a) and (b) show plots for all threshold values, ranging from 0-100% and the respective binary images created for each threshold value.).
Therefore it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Zhang in view of Popp by applying a threshold to reliably detect passage boundaries, that is taught by Fahimi, to make the invention that applies a threshold to the image so that the portal (passage) section is extracted as a distinct region (Fahimi, Fig. 3); thus one of ordinary skill in the art would have been motivated to combine the references to apply an effective thresholding method to extract the particular area that in turn leads to an estimation of portal boundaries (Fahimi, “Dynamic Thresholding” Section).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Garcia et al. (NPL "A Convolutional Neural Network Feature Detection Approach to Autonomous Quadrotor Indoor Navigation," 2019, hereafter referred to as Garcia) in view of Li et al. (NPL "UAV identifies passable paths in unknown indoor environments based on Deep-learning," 2022, hereafter referred to as Li) in further view of Yang et al. (NPL "Computer Vision-Based Door Detection for Accessibility of Unfamiliar Environments to Blind Persons," 2010, hereafter referred to as Yang).
Regarding Claim 19, Garcia teaches a method for real-time detection and tracking of potential passages in an environment (Section I Introduction, Section IV(C) Structural Feature Detection, Garcia teaches a prediction model that can reliably and consistently detect and locate structural features from a drone’s sole forward-looking camera at frame processing rates around 25 fps. Such a frame rate permits autonomous real-time navigation flights. Structural features include front-facing doors, side doors, and windows.), computing one or more bounding boxes for the one or more passages (Section II(A) System Architecture, Fig. 3, Garcia discloses processing the received video frame and publishing a container filled with metadata describing the number of structures detected along with a list of bounding box data. The list of bounding box data contains all the bounding boxes that were detected in the video frame. Each bounding box data contains the x and y coordinates of the top left most and bottom right most points of the bounding box.), and tracking the one or more bounding boxes between two or more frames of image data (Section IV(D) Distance Estimation Using SVM Regression Model, Section II(A) System Architecture, Garcia discloses placing a drone at various distances away from a structural feature of interest and recording the changes in the bounding box dimensions as the drone progressively moves closer to the structure. In an example, the drone was set 55 meters away from the structure and progressively moved closer until the drone was 2 meters away. The vision system was able to always detect and localize the feature at all distances in the range 55 m to 2 m. Types of structures include doorways, and front-doors. The examiner interprets that recording the changes in bounding box dimensions in each successive image encompasses the act of tracking the bounding box between the frames of image data.).
Garcia does not explicitly disclose computing semantic segmentation of one or more passages in one or more frames of image data; and wherein the boundary of each of the of the bounding boxes is computed based on edge detection and corner detection of the one or more passages.
Li is in the same field of art of identifying passages, including doors, in indoor environments. Further, Li teaches computing semantic segmentation of one or more passages in one or more frames of image data (Section II(B) Target detection selection, Li teaches distinguishing the door-only area from walls or other objects in the image. This requires segmentation of the image, and segmenting the area where the gate is located can also be called semantic segmentation.).
Therefore it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Garcia by performing semantic segmentation on the passages in the image, that is taught by Li, to make the invention that is able to distinguish passages from one another, such as doors from walls or other objects in the image (Li, Section II(B) Target detection selection); thus one of ordinary skill in the art would have been motivated to combine the references to ensure that the passage is outlined accurately and only includes areas that are actually part of the target door in the segmentation result (Li, Section II(B) Target detection selection).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art.
Garcia in view of Li does not explicitly disclose wherein the boundary of each of the of the bounding boxes is computed based on edge detection and corner detection of the one or more passages.
Yang is in the same field of art of identifying passages, such as doors, to support navigation through unfamiliar environments. Further, Yang discloses wherein the boundary of each of the of the bounding boxes is computed based on edge detection and corner detection of the one or more passages (Abstract, Section 3.2, Fig. 2, Yang teaches an image-based door detection algorithm based on doors’ general and stable features, edges and corners, rather than appearance features. Edges are extracted through Canny edge detection and a binary edge map is created. The corners are then extracted through the edge map based on local and global curvature properties by a corner detector. Four corners of a doorframe can be extracted by the corner detection algorithm regardless of occlusion. Fig. 2(a) depicts the four corner points C1-C4 with coordinates (xi, yi). The corners are connected by lines (L12, L23, L34, and L41). The examiner interprets that Fig. 1(a) depicts a bounding box, which is broadly defined in the art as a rectangular region defined by coordinates that encloses an object of interest within an image or video frame since the claim is silent to the specifications of the bounding box.).
Therefore it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Garcia in view of Li by generating a bounding box for the passage based on the detected edge and corner information, that is taught by Yang, to make the invention that is able to detect passages in an image frame regardless of variations of scales, colors, viewpoints, and light chances in an image (Yang, Abstract); thus one of ordinary skill in the art would have been motivated to combine the references to develop a more robust door/passage detector able to handle different environments, by combining both edge and corner features to characterize the geometric shape of the passage (Yang, Section 1.2).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art.
Allowable Subject Matter
Claims 3, 5-13, 15, 17, and 18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding Claim 3, No prior art teaches the feature of applying a regression output that detects passage edges in each bounding box.
In regards to Claim 5 and dependents, No prior art teaches processing the encoder output with a second decoder with a regression output.
In regards to Claim 17 and dependents, No prior art teaches wherein the tracking one or more points between frames of image data for each of the one or more detected passages in one or more frames of image data comprises: a) looping over existing passage descriptors to find the highest intersection over union (IoU) with each new passage detection; b) if there is no overlap between a new frame and a previous frame, then creating a new descriptor for the detected passage; c) if there is overlap between a new frame and a previous frame, then choosing the highest IoU as a new passage detection.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SYDNEY L BLACKSTEN whose telephone number is (571)-272-7651. The examiner can normally be reached 8:30am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Oneal Mistry can be reached at 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SYDNEY L BLACKSTEN/Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674