DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 01/17/2025 has been entered.
Status of the Claims
Claims 1-20 of U.S. Application No. 17/663,423 filed on 05/14/2022 were examined. Examiner filed a non-final rejection on 05/21/2024.
Applicant filed remarks and amendments on 08/21/2024. Claims 1-3, 5-7, 9-13, 15-17, and 19-20 are amended. Claims 1 – 20 were examined. Examiner filed a final rejection on 11/19/2024.
Applicant filed an RCE on 01/17/25. Claims 1-3, 6, 11-13, and 16 were amended. Claims 1-20 were examined. Examiner filed a non-final rejection on 04/24/2025.
Applicant filed remarks and amendments on 07/24/2025. Claim(s) 1, 2, 6, 9, 11, 12, 16, and 19 are amended. Claims 1 – 20 are presently pending and presented for examination.
Response to Arguments
Regarding the claim rejections under 35 USC 101: Applicant's arguments filed 07/24/2025 have been fully considered and they are persuasive. The previously given claim interpretations under 35 USC 101 are withdrawn.
Regarding the claim rejections under 35 USC 103: Applicant's arguments filed 07/24/2025 with respect to Ebrahimi et al. (US 20220066456 A1) in view of Ebrahimi et al. (US 9764472 B1) have been fully considered but they are not persuasive.
Regarding claims 1 and 11, applicant argues that the “combination of the cited references fails to teach or suggest collecting sensor data in the manner now recited in the amended claims, using a triplet as now defined in the amended claims…”.
However, the examiner respectfully disagrees this argument is not persuasive. Ebrahimi456 expressly discloses collecting positional sensor data as a continuous time series stream from autonomous vehicles operating in a warehouse environment.
Ebrahimi456, ¶ [0414]: “In some embodiments, the robot executes the path using markings on the floor that the processor of the robot detects based on sensor data collected by sensors of the robot. The processor uses sensor data to continuously detect and follow markings”
Moreover, Ebrahimi456 teaches forming triplets of positional data (current, inertial, and prior position) to detect turning/cornering events:
Ebrahimi456, ¶ [0342]: “Each field of view observes data at each time point t1, t(1+1), t(1+2), . . . , tn. Some embodiments implement a rectifying process wherein the observations captured in fields of view FOV1, FOV2, FOV3, . . . , FOVn of cameras c1, c2, c3, . . . , cn are bundled. Examples of different types of data that may be bundled include any of GPS data, IMU data, SFM data, laser range finder data, depth data, optical tracker data, odometer data, radar data, sonar data, etc..” Thus, the combination clearly teaches collecting time-series positional data and using positional triplets exactly as claimed.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Ebrahimi et al. (US 20220066456 A1) in view of Ebrahimi et al. (US 9764472 B1), hereinafter referred to as Ebrahimi456 and Ebrahimi472 respectively.
Regarding claims 1 and 11, Ebrahimi456 discloses A method for detecting events at nodes operating in a warehouse environment(“In some embodiments, the processor detects an edge based on a rate of change of depth readings collected by a sensor (e.g., depth sensor) or a rate of change of pixel intensity of pixels in an image. In embodiments, the processor may use various methods to detect an edge or any other feature that reduces the points against which the processor localizes the robot. For instance, different features extracted from images or from depth data may be used by the processor to localize the robot. In cases wherein depth data is used, the processor uses the distance between the robot and the surroundings (e.g., a wall, an object, etc.) at each angular resolution as a constraint that provides the position of the robot in relation to the surroundings. In embodiments, the robot and a measurement at a particular angle form a data pair. For example, each depth measurement taken at a particular angle and the robot form a data pair. For instance, a first single measurement at a particular angle and the robot form a data pair and a second single measurement at a particular angle and the robot from another data pair, and so on.” [0522]),
the method comprising:
Generating sensor data at a node comprising a forklift or an automated robot operating in the warehouse environment, wherein the sensor data includes positional data of the node in the warehouse environment, wherein the sensor data is collected as a time series data stream (“generating, by the processor of the robot, additional iterations of the map based on newly captured LIDAR data and newly captured movement data obtained as the robot performs coverage and traverses into new and undiscovered areas,” [0006] and “While several examples illustrate or describe two frames, wherein one image is taken and a second image is taken immediately after, the concepts described herein are not limited to being applied to two images and may be used for a series of images (e.g., video).” [0918]);
When the triplet indicates that the cornering event is detected, inputting the detected event into a model configured to determine whether the detected cornering event is a non-normative cornering event (“In some embodiments, the AP signal strength data collected by sensors of the robot are fed into the deep neural network model along with accurate LIDAR measurements” [0309] and “A third layer extracts low level features using techniques such as Fourier descriptors, edge detection techniques, corner detection techniques,” [0280]), wherein the model is trained only with normative sensor data form multiple nodes corresponding to normative cornering events, the normative sensor data including triplets corresponding to normative cornering events (“In an example of a neural network, a first layer receives input. A second layer extracts extreme low level features by detecting changes in pixel intensity and entropy. A third layer extracts low level features using techniques such as Fourier descriptors, edge detection techniques, corner detection techniques,” [0280] and “Some embodiments may merge various types of data into a data structure, clean the data, extract the converged data, encode the data to automatic encoders, use and/or store the data in the cloud, and if stored in the cloud, retrieve only the data needed for use locally at the robot or network level. Such merged data structures may be used by algorithms that remove outlines, algorithms that decide dynamic obstacle half-life or decay rate, algorithms that inflate troublesome obstacles, algorithms that identify where different types sensors act weak and when to integrate their readings (e.g., a sonar range finder acts poor where there are corners or sharp and narrow obstacles), etc.” [0310], wherein the non-normative event is detected when a loss of the model exceeds a threshold wherein the threshold is defined by a mean (p), a standard deviation (a), and a tolerance parameter z that satisfy the equation t = u+ z * a (“In some embodiments, the refining sample matcher algorithm may pass the few possible matching locations as a seed set to a full scan matcher algorithm. In some embodiments, the full scan matcher algorithm may choose a first seed as a match if the match score or probability of matching is above a predetermined threshold.” [1125] and “The processor may compare the error (y−(ax+b))2 of data points n1 to n2 to a threshold T1 and summates the number of errors below the threshold. The processor may then compute the difference between the number of points considered (n2−n1) and the number of data points with errors below threshold T1. If the difference is below a threshold T2, i.e., ((n2−n1)−Σn 1 n 2 (y−(ax+b))2<T1)<T2, then the processor assigns the data points to be a wall and otherwise assigns the data points to be an opening in the wall.” [1012]));
Performing a corrective action in substantially real time at the node when the detected event at the node is the non-normative cornering event, the corrective action including altering the movement of the node to prevent the non-normative cornering event from occurring and providing a notice of the non-normative cornering event (“ For instance, measurements for a LIDAR, depth camera, or CCD camera may be used to correct for drift caused by errors in the reading stream of a first type of sensing. While the method by which corrections are made may be dependent on the type of sensing, the overall concept of correcting an uncertainty caused by actuation using at least one other type of sensing remains the same.” [0422] and “In some embodiments, the response to a stuck condition may include driving the robot forward, and if the timeout threshold of the control action of driving the robot forward is exceeded, driving the robot backwards in an attempt to become unstuck.” [1118]).
Ebrahimi456 does not explicitly teach Performing event detection using at least a triplet of the positional data, the triplet of the positional data including position data, inertial data, one or more of load data and mast position data, wherein distances between the triplets is known and a cornering event is detected when an internal angle of the triplet is less than a threshold angle
However, Ebrahimi472 does teach Performing event detection using at least a triplet of the positional data, the triplet of the positional data including position data, inertial data, one or more of load data and mast position data, wherein distances between the triplets is known and a cornering event is detected when an internal angle of the triplet is less than a threshold angle (“In some embodiments, establishing the coverage path that accounts for all detected obstacles includes any of: calculating a first center-return path, where the first center-return path is defined by a number of substantially collision-free triangulated paths, and where the first center-return path corresponds with a maximum turning angle, a triangle measured size with respect to a tool size of the robotic device and distances to all detected obstacles; calculating a second center-return path, where the second center-return path is defined by a number of substantially collision-free isosceles triangulated paths, and where the second center-return path corresponds with a maximum turning angle, an isosceles triangle measured size with respect to the tool size of the robotic device and a distance to a first newly detected obstacle; and calculating a third center-return path, where the third center-return path is defined by a number of substantially collision-free point-to-point paths, and where the third center-return path corresponds with a maximum turning angle and a number of straight lines each having a measured size with respect to detected obstacles.” [Col.3 ln 30-50]). Both Ebrahimi456 and Ebrahimi472 teach methods for determining vehicle environment surroundings and events using positional data. However, Ebrahimi472 explicitly teaches Performing event detection using at least a triplet of the positional data, the triplet of the positional data including position data, inertial data, one or more of load data and mast position data, wherein distances between the triplets is known and a cornering event is detected when an internal angle of the triplet is less than a threshold angle.
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the robot detection method of Ebrahimi456 to also include Performing event detection using at least a triplet of the positional data, the triplet of the positional data including position data, inertial data, one or more of load data and mast position data, wherein distances between the triplets is known and a cornering event is detected when an internal angle of the triplet is less than a threshold angle, as taught by Ebrahimi472, with a reasonable expectation of success. Doing so improves safety and efficiency of robot or automated vehicle operation (With regard to this reasoning, see at least [Ebrahimi472, Col.3 ln 15-67]).
Regarding claims 2 and 12, Ebrahimi456 discloses The method of claim 1, wherein only positional data corresponding to the cornering event is input into the model (“In embodiments, the combination of the convolution layer, detector layer (i.e., ReLu), and pooling layer are referred to as the convolution layer (although each layer could be technically viewed as an independent layer). Therefore, in the figures included herein, some layers may not be shown. While pooling helps reach invariance, which is useful for detecting edges, corners and identifying objects, eyes, and faces, it suppresses properties that may help detect translational or angular displacement. Therefore, in embodiments, it is necessary to pool over the output of separately parametrized convolutions and train the network on where invariance is needed and where it is harmful. In one case, invariance is required to distinguish a number 5 based on, for example, edge detection. In another case, invariance may be harmful, wherein the goal is to determine a change in position of the robot. If the objective is to distinguish the number 5, invariance is needed, however, if the objective is to use the number 5 to determine how the robot changed in position and heading, invariance jeopardizes the application. The network may conclude that the number 5 at a current time is observed to be larger in size and therefore the robot is closer to the number 5 or that the number 5 at a current time is distorted and therefore the robot is observing the number 5 from a different angle.” [0327]).
Regarding claims 3 and 13, Ebrahimi456 discloses The method of claim 2, further comprising processing the sensor data to identify the positional data the cornering event(“In one example a corner detected by a processor of a robot based on sensor data may be used to localize the robot. For instance, a camera positioned on the robot captures a first image of the environment and detects a corner at a first time point t.sub.0. At a second time point t.sub.1, the camera captures a second image and detects a new position of the corner. The difference in position between the position of corner in the first image and the second image may be used in determining an amount of movement of the robot and localization. In some embodiments, the processor detects the corner based on change in pixel intensity, as the rate of change in pixel intensity increases in the three directions that intersect to form the corner.” [0437]).
Regarding claims 4 and 14, Ebrahimi456 discloses The method of claim 2, further comprising performing checks on the sensor data (“In some embodiments, the processor may use data binning to reduce the effects of minor observation errors and/or reduce the amount of data to be processed. The processor may replace original data values that fall into a given small interval, i.e. a bin, by a value representative of that bin (e.g., the central value). In image data processing, binning may entail combing a cluster of pixels into a single larger pixel, thereby reducing the number of pixels. This may reduce the amount data to be processor and may reduce the impact of noise.” [0422]),
the checks including determining that the sensor data corresponds to real world specifications (“In some embodiments, the processor may check whether a next navigation goal (e.g., a path to a particular point) is safe using the local map. A next navigation goal may be considered safe if it is within the local map and at a safe distance from local obstacles, is in an area outside of the local map, or is in an area labelled as unknown. In some embodiments, wherein the next navigation goal is unsafe, the processor may perform a wave search from the current location of the robot to find a safe navigation goal that is inside of the local map and may plan a path to the new navigation goal.” [0938]),
discarding data representing periods of non-movement (“In some embodiments, the processor uses information theory to avoid processing images that do not bear information. This step in the process is the key frame/image selection step. In embodiments, the processor may remove blurred images due to motion, lighting issues, etc. to filter out undesired images. In some embodiments, discarded images may be sent and used elsewhere for more in depth processing. For example, the discarded images may be sent to higher up processors, GPUs, the cloud, etc. After pruning unwanted images, the processor may determine using two consecutive images how much the camera positioned on the robot moved (i.e., or otherwise how much the robot moved) and how much the tennis ball moved. The processor may infer where the ball will be located next by determining the heading angular and linear speed and momentum of the ball, geo-characteristics of the environment, rules of motion of the ball, and possible trajectories.” [0447]),
discarding data with high levels of noise that cannot be normalized (“The process is repeated and less outliers are found in this case as there are less data points with distances 4104 falling outside of region 4105. In some embodiments, the processor may continue to choose another two data points and repeat the process until a minimum number of outliers is found or the number of outliers or the ratio of outliers to inliers is below a predetermined threshold. In some embodiments, there may be too may data points within the set of data points to select data points in sets of two. In some embodiments, the processor may probabilistically determine the number of data points to select and check based on the accuracy or minimum probability required. For example, the processor may iterate the method 20 times to achieve a 99% probability of success. Any of the methods and techniques described may be used independently or sequentially, one after another, or may be combined with other methods and may be applied in different orders.” [0986]),
discarding data where position data is not available (“In case of the LIDAR being covered (i.e., not available), the processor of the robot may use gyroscope data to continue mapping and covering hard surfaces since a gyroscope performs better on hard surfaces.” [0587]),
and/or amending the positional data in selected instances where there is a gap in a time threshold of jumps in the position data (“In VR, MR mode, the user may move the camera using hand movements or gestures. Replay mode is another navigation mode users may use, wherein a replay of the robot's coverage in 3D may be viewed. In this case, a virtual camera is moves along the paths the robot has already completed. The user has some control over the replay by forwarding, rewinding, adjusting a speed, time jumping, playing, pausing, or even changing the POV of the replay. For example, if sensors of the robot are facing forward as the robot completes the path, during the replay, the user may change their POV such that they face towards the sides or back of the robot while the camera still follows along the path of the robot.” [0374]).
Regarding claims 5 and 15, Ebrahimi456 discloses The method of claim 1, further wherein the model is an autoencoder and non- normative events are detected based on the loss associated with the event input into the model (“In some embodiments, the processor may use run-length encoding (RLE), a form of lossless data compression, to store runs of data (consecutive data elements with the same data value) as a single data value and count instead of the original run. For example, an image containing only black and white may have many long runs of white pixels and many short runs of black pixels. A single row in the image may include 67 characters, each of the characters having a value of 0 or 1 to represent either a white or black pixel. However, using RLE the single row of 67 characters may be represented by 12W1B12W3B24W1B14 W, only 18 characters which may be interpreted as a sequence of 12 white pixels, 1 black pixel, 12 white pixels, 3 black pixels, 24 white pixels, 1 black pixel, and 14 white pixels. In embodiments, RLE may be expressed in various ways depending on the data properties and compression algorithms used.” [0263]).
Regarding claims 6 and 16, Ebrahimi456 discloses The method of claim 5, wherein the multiple nodes includes the node (“In some embodiments, time dependent data may include certain attributes. For instance, all data may not be collected before a classification tree is generated; all data may not be available for revisiting spontaneously; previously unseen data may not be classified; all data is real-time data; data assigned to a node may be reassigned to an alternate node; and/or nodes may be merged and/or split.” [0283]).
Regarding claims 7 and 17, Ebrahimi456 discloses The method of claim 6, further comprising performing checks on the sensor data from the multiple nodes and identifying normative cornering data from the sensor data such that the autoencoder is trained using only the normative cornering data, the cornering data including triplets of positional data (“In some embodiments, the processor uses an autoencoder to train a classifier. In some embodiments, unlabeled data is gathered. In some embodiments, the processor trains a deep autoencoder using data including labelled and unlabeled data. Then, the processor trains the classifier using a portion of that data, after which the processor then trains the classifier using only the labelled data. The processor cannot put each of these data sets in one layer and freeze the reused layers. This generative model regenerates outputs that are reasonably close to training data.” [0278]).
Regarding claims 8 and 18, Ebrahimi456 discloses The method of claim 7, wherein training the autoencoder with only the normative cornering data allows unsafe cornering events to be inferred based on a loss of the autoencoder (“In some embodiments, the processor uses an autoencoder to train a classifier. In some embodiments, unlabeled data is gathered. In some embodiments, the processor trains a deep autoencoder using data including labelled and unlabeled data. Then, the processor trains the classifier using a portion of that data, after which the processor then trains the classifier using only the labelled data. The processor cannot put each of these data sets in one layer and freeze the reused layers. This generative model regenerates outputs that are reasonably close to training data.” [0278] and “The processor may restrict zones to being rectangular (or having some other defined number of vertices or sides) and may restrict the robot to entering a zone at a corner and to driving a serpentine routine (or other driving routine) in either x- or y-direction such that the trajectory ends at another corner of the zone. The cost associated with a particular division of an environment and order of zone coverage may be computed as the sum of the distances of the serpentine path travelled for coverage within each zone and the sum of the distances travelled in between zones (corner to corner).”[1164]).
Regarding claims 9 and 19, Ebrahimi456 discloses The method of claim 8, wherein the loss is associated with a reconstruction error, wherein the event is a non-normative event when the reconstruction error exceeds the threshold (“In some embodiments, some data, such as old data or obsolete data, may be discarded. For instance, observation data of a home that has been renovated may be obsolete or some data may be too redundant to be useful and may be discarded. In some embodiments, data collected and/or used within the past 90 days is kept intact. In some embodiments, data collected and/or used more than two years ago may be discarded. In some embodiments, the data collected and/or used more than 90 days ago but before two years ago that does not show statistically significant difference from their counterparts may be discarded. In some embodiments, autoencoders with a linear activation and a cost function (e.g., mean squared error) may be used to reconstruct data.” [0270]).
Regarding claims 10 and20, Ebrahimi456 discloses The method of claim 1, wherein the model is configured to consider data triplets when determining whether the detected event is non-normative (“It is important to note that it can be a doublet (two elements) or triplet (three elements); the number of elements is not related to the number of rays for which it corrects. In other words, an achromatic lens designed for visible wavelengths corrects for red and blue, independent of it being a doublet or triplet configuration. However apochromatic lenses are designed to bring three colors into focus in the same plane.” [0645]),
wherein, the triplets are each associated with an internal angle and an external angle (“The processor may, for example, receive and process data from internal or external sensors, execute commands based on data received, control motors such as wheel motors, map the environment, localize the robot, determine division of the environment into zones, and determine movement paths.” [0238]) that determine whether the event is a cornering event (“The robot may first turns 90 degrees and the processor may make a decision based on the new incoming sensor information. As the robot navigates within the environment, the processor creates a map based on confirmed spaces. The robot may follow the perimeters of the obstacles it encounters and other geometries to find and cover spaces that may have possibly been missed. When finished coverage, the robot may go back to the starting point.” [0576]] and “In some embodiments, the processor may use an extended Kalman filter such that correspondences are incrementally updated. This may be applied to both depth readings and feature readings in scenarios wherein the FOV of the robot is limited to a particular angle around the 360 degrees perimeter of the robot and scenarios wherein the FOV of the robot encompasses 360 degrees through combination of the FOVs of complementary sensors positioned around the robot body or by a rotating LIDAR device. The SLAM algorithms used by the processor may use data from solid state sensors of the robot and/or a 360 degrees LIDAR with an internally rotating component positioned on the robot. The FOV of the robot may be increased by mechanically overlapping the FOV of sensors positioned on the robot. In an example of overlapping FOVs of cameras positioned on a robot, the overlap of FOVs extends the horizontal FOV of the robot. In another example, the overlapping FOVs of cameras positioned on the robot 7702 extends the vertical FOV of the robot. In some cases, the robot includes a set of sensors that are used concurrently to generate data with improved accuracy and more dimensions. For instance, the robot may include a two-dimensional LIDAR and a camera, which when used in tandem generates three-dimensional data.” [0348]).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AHMED ALKIRSH whose telephone number is (703) 756-4503. The examiner can normally be reached M-F 9:00 am-5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, FADEY JABR can be reached on (571) 272-1516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
AHMED ALKIRSHExaminer, Art Unit 3668
/Fadey S. Jabr/Supervisory Patent Examiner, Art Unit 3668