Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1- 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The limitations, under their broadest reasonable interpretation, cover mental process (concept performed in a human mind, including as observation, evaluation, judgment, opinion). The claims recite a method of creating a digital twin of a physical environment. This judicial exception is not integrated into a practical application because the steps do not add meaningful limitations to be considered specifically applied to a particular technological problem to be solved. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the steps of the claimed invention can be done mentally and no additional features in the claims would preclude them from being performed as such.
According to the USPTO guidelines, a claim is directed to non-statutory subject matter if:
STEP 1: the claim does not fall within one of the four statutory categories of invention (process, machine, manufacture or composition of matter), or
STEP 2: the claim recites a judicial exception, e.g., an abstract idea, without reciting additional elements that amount to significantly more than the judicial exception, as determined using the following analysis:
STEP 2A (PRONG 1): Does the claim recite an abstract idea, law of nature, or natural phenomenon?
STEP 2A (PRONG 2): Does the claim recite additional elements that integrate the judicial exception into a practical application?
STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
Using the two-step inquiry, it is clear that claims directed to an abstract idea as shown below:
STEP 1: Do the claims fall within one of the statutory categories?
YES. Claim 1 is directed to a method, i.e., process, claim 14 is directed to a device, i.e., a machine and claim 18 is directed to a device, i.e., a machine.
STEP 2A (PRONG 1): Is the claim directed to a law of nature, a natural phenomenon or an abstract idea?
YES, the claims are directed toward a mental process (i.e., abstract idea).
With regard to STEP 2A (PRONG 1), the guidelines provide three groupings of subject matter that are considered abstract ideas:
Mathematical concepts – mathematical relationships, mathematical formulas or equations, mathematical calculations;
Certain methods of organizing human activity – fundamental economic principles or practices (including hedging, insurance, mitigating risk); commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations); managing personal behavior or relationships or interactions between people (including social activities, teaching, and following rules or instructions); and
Mental processes – concepts that are practicably performed in the human mind (including an observation, evaluation, judgment, opinion).
The method in claims 1, 14 and 18 comprise a mental process that can be practicably performed in the human mind therefore, an abstract idea.
Claims 1 and 14 recite:
Processing for an object in the IF using an object library:
identify an object class
determine a probability of recognition of the object
perform pose estimation for the object, and
determine coordinates of the object within the IF;
determining a distance to the object by at least one of using an external sensor or triangulating with at least two video sensors having known fields of view and distances between sensors;
calculating, coordinates of the object within the physical space;
repeating the acquiring, processing, determining, and calculating for a next image frame (IF+1);
performing, prefiltering to exclude false positives of the object or the coordinates of the object using IF and IF+1 as prefiltered object information;
(a human can visually perform object detection on image as a mental process as an abstract idea); and
Claim 18 recites:
determine first positional data for an object in a first image frame (IF) captured by the first image sensor;
process the first IF for an object library associated with a digital twin, and configured to, for an object in the first IF using the object library and the first positional data, determine object information in the first IF including an object class, an object direction, a pose estimation, and object coordinates;
determine second positional data for an object in a second image frame (IF) captured by the second image sensor;
process the second IF for the object library, and configured to, for an object in the second IF using the object library and the second positional data, determine object information in the second IF including an object class, an object direction, a pose estimation, and object coordinates;
aggregate the object information of the first camera and the second camera as aggregated object information, and
synchronize the aggregated object information as synchronize aggregated object information;
(a human can visually perform object detection on image and process detected data as a mental process as an abstract idea); and
These limitations, as drafted, is a simple process that, under their broadest reasonable interpretation, covers performance of the limitations in the mind or by a human. The Examiner notes that under MPEP 2106.04(a)(2)(III), the courts consider a mental process (thinking) that “can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). As the Federal Circuit explained, "methods which can be performed mentally, or which are the equivalent of human mental work, are unpatentable abstract ideas the ‘basic tools of scientific and technological work’ that are open to all.’" 654 F.3d at 1371, 99 USPQ2d at 1694 (citing Gottschalk v. Benson, 409 U.S. 63, 175 USPQ 673 (1972)). See also Mayo Collaborative Servs. v. Prometheus Labs. Inc., 566 U.S. 66, 71, 101 USPQ2d 1961, 1965 ("‘[M]ental processes[] and abstract intellectual concepts are not patentable, as they are the basic tools of scientific and technological work’" (quoting Benson, 409 U.S. at 67, 175 USPQ at 675)); Parker v. Flook, 437 U.S. 584, 589, 198 USPQ 193, 197 (1978) (same).
The courts do not distinguish between mental processes that are performed entirely in the human mind and mental processes that require a human to use a physical aid (e.g., pen and paper or a slide rule) to perform the claim limitation. See, e.g., Benson, 409 U.S. at 67, 65, 175 USPQ at 674-75, 674 (noting that the claimed "conversion of [binary-coded decimal] numerals to pure binary numerals can be done mentally," i.e., "as a person would do it by head and hand."); Synopsys, Inc. v. Mentor Graphics Corp., 839 F.3d 1138, 1139, 120 USPQ2d 1473, 1474 (Fed. Cir. 2016) (holding that claims to a mental process of "translating a functional description of a logic circuit into a hardware component description of the logic circuit" are directed to an abstract idea, because the claims "read on an individual performing the claimed steps mentally or with pencil and paper").
Nor do the courts distinguish between claims that recite mental processes performed by humans and claims that recite mental processes performed on a computer. As the Federal Circuit has explained, "[c]ourts have examined claims that required the use of a computer and still found that the underlying, patent-ineligible invention could be performed via pen and paper or in a person’s mind." Versata Dev. Group v. SAP Am., Inc., 793 F.3d 1306, 1335, 115 USPQ2d 1681, 1702 (Fed. Cir. 2015). See also Intellectual Ventures I LLC v. Symantec Corp., 838 F.3d 1307, 1318, 120 USPQ2d 1353, 1360 (Fed. Cir. 2016) (‘‘[W]ith the exception of generic computer-implemented steps, there is nothing in the claims themselves that foreclose them from being performed by a human, mentally or with pen and paper.’’); Mortgage Grader, Inc. v. First Choice Loan Servs. Inc., 811 F.3d 1314, 1324, 117 USPQ2d 1693, 1699 (Fed. Cir. 2016) (holding that computer-implemented method for "anonymous loan shopping" was an abstract idea because it could be "performed by humans without a computer").
Because both product and process claims may recite a "mental process", the phrase "mental processes" should be understood as referring to the type of abstract idea, and not to the statutory category of the claim. The courts have identified numerous product claims as reciting mental process-type abstract ideas, for instance the product claims to computer systems and computer-readable media in Versata Dev. Group. v. SAP Am., Inc., 793 F.3d 1306, 115 USPQ2d 1681 (Fed. Cir. 2015).
As such, a person could perform visual object detection either mentally or using a pen and paper. The mere nominal recitation that the various steps are being executed by one or more hardware processors (e.g. processing unit) does not take the limitations out of the mental process grouping. Thus, the claims recite a mental process.
If a claim limitation, under its broadest reasonable interpretation, covers performance of a mental step which could be performed with a simple tool such as a pen and paper, then it falls within the “mental steps” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
STEP 2A (PRONG 2): Does the claim recite additional elements that integrate the judicial exception into a practical application?
NO, the claims do not recite additional elements that integrate the judicial exception into a practical application.
With regard to STEP 2A (prong 2), whether the claim recites additional elements that integrate the judicial exception into a practical application, the guidelines provide the following exemplary considerations that are indicative that an additional element (or combination of elements) may have integrated the judicial exception into a practical application:
an additional element reflects an improvement in the functioning of a computer, or an improvement to other technology or technical field;
an additional element that applies or uses a judicial exception to affect a particular treatment or prophylaxis for a disease or medical condition;
an additional element implements a judicial exception with, or uses a judicial exception in conjunction with, a particular machine or manufacture that is integral to the claim;
an additional element effects a transformation or reduction of a particular article to a different state or thing; and
an additional element applies or uses the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception.
While the guidelines further state that the exemplary considerations are not an exhaustive list and that there may be other examples of integrating the exception into a practical application, the guidelines also list examples in which a judicial exception has not been integrated into a practical application:
an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea;
an additional element adds insignificant extra-solution activity to the judicial exception; and
an additional element does no more than generally link the use of a judicial exception to a particular technological environment or field of use.
Claims 1- 20 do not recite any of the exemplary considerations that are indicative of an abstract idea having been integrated into a practical application.
Claims 1 and 14 recites:
AI video sensor (instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea).
claims 1 and 14 recite:
acquiring a real-time image frame (IF) by at least one AI video sensor positioned in the physical environment;
(adding insignificant extra-solution activity to the judicial exception, e.g., mere data gathering in conjunction with a law of nature or abstract idea);
transmitting, by the AI video sensor, the prefiltered object information as a compact data message to a remote backend for dynamically reconstructing the digital twin, the compact message including: the object class, an object direction, the pose estimation, a timestamp, an AI video sensor ID, and the coordinates of the object.
(adding insignificant extra-solution activity to the judicial exception, e.g., mere data outputting in conjunction with a law of nature or abstract idea);
Claim 8 recites:
a first image sensor positioned in a physical space;
a first positioning subsystem ;
a first camera configured with a convolutional neural network (CNN) trained based on an object library
a second image sensor positioned in the physical space;
a second positioning system
a second camera
and a global server (instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea).
claim 18 recites:
receive, over a high bandwidth interface, the object information of the first camera and the second camera, (adding insignificant extra-solution activity to the judicial exception, e.g., mere data gathering in conjunction with a law of nature or abstract idea);
transmit the synchronized aggregated object information to the digital twin.
(adding insignificant extra-solution activity to the judicial exception, e.g., mere data outputting in conjunction with a law of nature or abstract idea);
claims 1, 14 and 18 recite:
a convolutional neural network (CNN)
the one or more trained machine learning models represent no more than mere instructions to apply the judicial exception on a computer OR merely uses the computer as a tool to perform an abstract idea. See MPEP 2106.05(f)).
The “one or more machine learning models” which appears as recited in the claims further stands in to automate a human mental process using a generic machine learning models that has not been improved by the applicant.
These limitations are recited at a high level of generality (i.e. as a general action or change being taken based on the results of the acquiring step) and amounts to mere post solution actions, which is a form of insignificant extra-solution activity. Further, the claims are claimed generically and are operating in their ordinary capacity such that they do not use the judicial exception in a manner that imposes a meaningful limit on the judicial exception. Accordingly, even in combination, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
With regard to STEP 2B, whether the claims recite additional elements that provide significantly more than the recited judicial exception, the guidelines specify that the pre-guideline procedure is still in effect. Specifically, that examiners should continue to consider whether an additional element or combination of elements:
adds a specific limitation or combination of limitations that are not well-understood, routine, conventional activity in the field, which is indicative that an inventive concept may be present; or
simply appends well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception, which is indicative that an inventive concept may not be present.
With regard to (2b) the Guidance provided the following examples of limitations that may be enough to qualify as “significantly more" when recited in a claim with a judicial exception:
Improvement to another technology or technical field
Improvement to functioning of computer itself and/or applying the judicial exception with, or by use of, a particular machine
Effecting a transformation or reduction of a particular article to a different state or thing.
Adding a specific limitation other that what is well understood, routine and conventional in the field, or adding unconventional steps that confine the claim to a particular useful application
Meaningful limitation beyond generally linking the use of an abstract idea to a particular technological environment.
The Guidance further set forth limitations that were found not to be enough to qualify as “significantly more” when recited in a claim with a judicial exception include:
Adding words to “apply it” (or an equivalent) with the judicial exception or mere instructions to implement abstract ideas on a computer
Simply appending well-understood, routine and conventional activities previously known to the industry specified at a high level of generality to the judicial exception, e.g. a claim to an abstract idea requiring no more than a generic
Computer to perform generic computer functions that are well -understood, routine and conventional activities previously known to the industry.
Adding insignificant extra-solution activity to the judicial exception, e.g. mere data gathering in conjunction with a law of nature or abstract idea
Generally linking the use of the judicial exception to a particular technological environment or field of use.
Claims 1- 20 do not recite any additional elements that are not well-understood, routine or conventional.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The above identified additional computer components, using instructions to apply the judicial exception, are merely generic computer components that are well-known, routine, and conventional as is evidenced by Bancorp Services v. Sun Life (Fed. Cir. 2012) and Alice Corp. v. CLS Bank (2014).
claims 1 and 14 recite:
acquiring a real-time image frame (IF) by at least one AI video sensor positioned in the physical environment;
(adding insignificant extra-solution activity to the judicial exception, e.g., mere data gathering in conjunction with a law of nature or abstract idea);
transmitting, by the AI video sensor, the prefiltered object information as a compact data message to a remote backend for dynamically reconstructing the digital twin, the compact message including: the object class, an object direction, the pose estimation, a timestamp, an AI video sensor ID, and the coordinates of the object.
(adding insignificant extra-solution activity to the judicial exception, e.g., mere data outputting in conjunction with a law of nature or abstract idea);
claims 1, 14 and 18 recite:
a convolutional neural network (CNN)
a convolutional neural network (CNN) represent no more than mere instructions to apply the judicial exception on a computer OR merely uses the computer as a tool to perform an abstract idea. See MPEP 2106.05(f)).
The “a convolutional neural network (CNN)” which appears as recited in the claims further stands in to automate a human mental process using a generic machine learning models that has not been improved by the applicant.
Thus, since claims 1, 14 and 18 are: (a) directed toward an abstract idea, (b) do not recite additional elements that integrate the judicial exception into a practical application, and (c) do not recite additional elements that amount to significantly more than the judicial exception, claims 1, 14 and 18 are not eligible subject matter under 35 U.S.C 101. Similar analysis is made for the dependent claims 2- 13, 15- 17 and 19- 20 and the dependent claims are similarly identified as: being directed towards an abstract idea, not reciting additional elements that integrate the judicial exception into a practical application, and not reciting additional elements that amount to significantly more than the judicial exception.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 13 and 14 are rejected under 35 USC 103 as being unpatentable over CN114155299 in view of Doumbouya et al. (US2018/0157916), AOKI (US 2012/0236124) and Beard et al. (20170300759).
With respect to claim 1, CN114155299 teaches acquiring a real-time image frame (IF) by at least one AI video sensor positioned in the physical environment (page 7, 10th para. S101. Acquire a surveillance video stream of the first architectural scene from a first photographing device installed in a first architectural scene, and acquire image frames to be processed frame by frame from the surveillance video stream);
Processing the IF using a convolutional neural network (CNN), for an object in the IF using an object library on the AI video sensor (page 8, 6th para., the YOLOv5 target detection algorithm may be used to perform the two-dimensional target detection step in step S102):
identify an object class (page 8, 5th para., the two-dimensional target detection result at least includes: the category to which each entity object belongs),
determine a probability of recognition of the object (page 8, 5th para., The target detection result may also include: the confidence level of the category to which each detected entity object belongs),
perform pose estimation for the object (page 8, 13th para., the three-dimensional pose detection model is used to predict the entity position and entity direction of each entity object in the image frame to be processed in the first building scene.), and
determine coordinates of the object within the IF (page 6 2nd para. predict the entity position and entity direction of each of the entity objects in the to-be-processed image frame);
calculating, coordinates of the object within the physical space;
repeating the acquiring, processing, determining, and calculating for a next image frame (IF+1) (page 7, 5th para. obtain the image frames to be processed frame by frame; perform two-dimensional target detection on each entity object included in the image frames to be processed);
transmitting, the prefiltered object information as a compact data message to a remote backend for dynamically reconstructing the digital twin, the compact message including: the object class, an object direction, the pose estimation, a timestamp, an AI video sensor ID, and the coordinates of the object. (page 18, para 15, communication connection may be through some communication interfaces, indirect coupling or communication connection of systems or units).
CN114155299 does not teach expressly that process image locally by the at least one AI video sensor; determining, by the AI video sensor, a distance to the object by at least one of using an external sensor or triangulating with at least two video sensors having known fields of view and distances between sensors; performing, by the AI video sensor, prefiltering to exclude false positives of the object or the coordinates of the object using IF and IF+1 as prefiltered object information;
Doumbouya et al. teach generating chip data from camera and transmitted (Fig. 4, para [0170], The CNNs are used in the depicted example embodiment to process chips (404) generated by the camera(108); para [0151] and [0152], example metadata of an Object Profile 702 with Chip 404; The Data 710 in Object Profile 702 and Object Profile 704 has, for example, content including time stamp, frame number, resolution in pixels by width and height of the scene, segmentation mask of this frame by width and height in pixels and stride by row width in bytes, classification (person, vehicle, other), confidence by percent of the classification, box (bounding box surrounding the profiled object) by width and height in normalized sensor coordinates, image width and height in pixels as well as image stride (row width in bytes), segmentation mask of image, orientation, and x & y coordinates of the image box.).
At the time of effective filing, it would have been obvious to a person of ordinary skill in the art to process image locally in camera and transmit metadata in the method of CN114155299.
The suggestion/motivation for doing so would have been that to optimize resources utilization trough distributed processing and for the bandwidth and server storage efficiency.
AOKI teaches determining, by the AI video sensor, a distance to the object by at least one of using an external sensor or triangulating with at least two video sensors having known fields of view and distances between sensors; (Fig. 1, para [0013][0014], distance d)
At the time of effective filing, it would have been obvious to a person of ordinary skill in the art to determine distance using triangulating with at least two video sensors in the method of CN114155299.
The suggestion/motivation for doing so would have been that using well-known method to accurately determine distance
Beard et al. teach performing, by the AI video sensor, prefiltering to exclude false positives of the object or the coordinates of the object using IF and IF+1 as prefiltered object information (para [0058], recursive-RANSAC algorithm over multiple frames in order to filter out false positives.)
At the time of effective filing, it would have been obvious to a person of ordinary skill in the art to filter out false positive in the method of CN114155299.
The suggestion/motivation for doing so would have been that to minimize tracking error in object detection.
Therefore, it would have been obvious to combine Doumbouya et al., AOKI and Beard et al. with CN114155299 to obtain the invention as specified in claim 1.
With respect to claim 13, Beard et al. teach the prefiltering further includes filtering the object information based on predefined rules (para [0058], recursive-RANSAC algorithm).
Claim 14 is rejected as same reason as claim 1 above.
Claims 3, 4, 5 and 7 are rejected under 35 USC 103 as being unpatentable over CN114155299 in view of Doumbouya et al. (US2018/0157916), AOKI (US 2012/0236124) and Beard et al. (20170300759) and in further view of Rudin et al. (US 2011/0169946).
With respect to claim 3, CN114155299, Doumbouya et al., AOKI and Beard et al. teach all the limitations of claim 1 as applied above from which claim 4 respectively depend.
CN114155299, Doumbouya et al., AOKI and Beard et al. do not teach expressly calculating the coordinates of the object within the physical space comprises using a global navigation satellite system (GNSS) receiver connected and synchronized with the AI video sensor, allowing determination of the position and orientation relative to the known position of one or more satellites at a strictly defined moment in time.
Rudin et al. teach calculating the coordinates of the object within the physical space comprises using a global navigation satellite system (GNSS) receiver connected and synchronized with the AI video sensor, allowing determination of the position and orientation relative to the known position of one or more satellites at a strictly defined moment in time (Fig. 3, para [0020]-[0024], O1, (position of camera) based on GPS and X (position of point P in the O coordinate system) based on parameters of camera at O1; [0004], The GPS information can be associated with the captured image via a time stamp of both the GPS information and the image or the UPS image can simply be embedded in the digital data of the photograph).
At the time of effective filing, it would have been obvious to a person of ordinary skill in the art to determine position of object in the image based on GPS information synchronized with camera in the method of CN114155299, Doumbouya et al., AOKI and Beard et al.
The suggestion/motivation for doing so would have been that to accurately determine position of object with known method,.
Therefore, it would have been obvious to combine Rudin et al. with Doumbouya et al., AOKI and Beard et al. and CN114155299 to obtain the invention as specified in claim 3.
With respect to claim 4, Rudin et al. teach calculating the coordinates of the object within the physical space comprises a determination using topographic references to the terrain, including by at least one of a map, a compass, a chronometer, a barometer, a sextant, an external GNSS receiver , an analysis of the location of known objects in the IF, or an operator command ((Fig. 3, para [0020]-[0024], O1, (position of camera) based on GPS and X (position of point P in the O coordinate system) based on parameters of camera at O1; [0004], The GPS information can be associated with the captured image via a time stamp of both the GPS information and the image or the UPS image can simply be embedded in the digital data of the photograph).
With respect to claim 5, Rudin et al. teach calculating the coordinates of the object within the physical space comprises applying a vectorized map to the IF ((Fig. 3, para [0020]-[0024], O1, (position of camera) based on GPS and X (position of point P in the O coordinate system) based on parameters of camera at O1; [0004], The GPS information can be associated with the captured image via a time stamp of both the GPS information and the image or the UPS image can simply be embedded in the digital data of the photograph, equation [8]).
With respect to claim 7, Rudin et al. synchronizing a plurality of AI video sensors and with respective external sensors for determining the distance to an object, including at least one of: applying a GNSS receiver connected and synchronized with the plurality of AI video sensors, to determine position and orientation relative to the known position of one or more satellites at a strictly defined point in time, hardware synchronization by a common synchronization pulse via general-purpose input/output (GPIO) or sending a message via a cross-board interface, or connecting to an Ethernet network using Network Time Protocol (NTP) ((Fig. 3, para [0020]-[0024], O1, (position of camera) based on GPS and X (position of point P in the O coordinate system) based on parameters of camera at O1; [0004], The GPS information can be associated with the captured image via a time stamp of both the GPS information and the image or the UPS image can simply be embedded in the digital data of the photograph, equation [1]).
Claims 8-12, 16 and 17 are rejected under 35 USC 103 as being unpatentable over CN114155299 in view of Doumbouya et al. (US2018/0157916), AOKI (US 2012/0236124) and Beard et al. (20170300759) and in further view of Langley et al. (US Patent 11,030,892).
With respect to claim 8, CN114155299, Doumbouya et al., AOKI and Beard et al. teach all the limitations of claim 1 as applied above from which claim 4 respectively depend.
CN114155299, Doumbouya et al., AOKI and Beard et al. do not teach expressly the AI video sensor is configured to selectively transmit information about the object based on predefined criteria.
Langley et al. teach the AI video sensor is configured to selectively transmit information about the object based on predefined criteria(Fig. 5, col. 11 line 19- col. 14 line 24; flow chart for an exemplary method 500 for capturing and filtering images by the camera system 101; ref label 545, Filter image(s) using neural network(s) The neural network algorithms may be applied by the processor 230 in order to detect objects 120 within the images and to assign confidence scores to objects; ref label 550, Remove image(s) with low confidence score(s); ref label 555, Retain/store image(s) with high confidence score(s); ref label 565, Transmit retained image(s) with high confidence score(s) over communications network with modem).
At the time of effective filing, it would have been obvious to a person of ordinary skill in the art to selectively transmit information about the object in the method of CN114155299, Doumbouya et al., AOKI and Beard et al.
The suggestion/motivation for doing so would have been that to save resource.
Therefore, it would have been obvious to combine Langley et al. with Doumbouya et al., AOKI and Beard et al. and CN114155299 to obtain the invention as specified in claim 8.
With respect to claim 9, Langley et al. teach the predefined criteria includes object importance (The neural network algorithms may be applied by the processor 230 in order to detect objects 120 within the images and to assign confidence scores to objects).
With respect to claim 10, Langley et al. teach the AI video sensor transmits object information when the object changes position, appears or disappears in the frame, or changes pose (Fig. 5, ref label 505 detect motion).
With respect to claim 11, Langley et al. teach upon authorization, transmitting, by the AI video sensor, encrypted image frames or sequences of image frames, in addition to the compact message (Fig. 5, ref label 555 compress as needed).
With respect to claim 12, Langley et al. teach the prefiltering includes tracking mutual positions and relative movements of objects from frame to frame to filter out errors due to at least one of: overlapping objects from different angles, incorrect object type or position determination due to CNN failures, or glare in the image frame (Fig. 5, ref label 525 compress as needed).
With respect to claim 16, CN114155299, Doumbouya et al., AOKI and Beard et al. teach all the limitations of claim 14 as applied above from which claim 16 respectively depend.
CN114155299, Doumbouya et al., AOKI and Beard et al. do not teach expressly prefiltering to exclude false positives includes modeling the coordinates of the object using an initial set of frames to forecast an object trajectory and excluding the prefiltered object information when a detected object trajectory does not match the forecast.
Langley et al. teach prefiltering to exclude false positives includes modeling the coordinates of the object using an initial set of frames to forecast an object trajectory and excluding the prefiltered object information when a detected object trajectory does not match the forecast (Fig. 5, col. 11 line 19- col. 14 line 24; flow chart for an exemplary method 500 for capturing and filtering images by the camera system 101; ref label 545, Filter image(s) using neural network(s) The neural network algorithms may be applied by the processor 230 in order to detect objects 120 within the images and to assign confidence scores to objects; ref label 550, Remove image(s) with low confidence score(s); ref label 555, Retain/store image(s) with high confidence score(s); ref label 565, Transmit retained image(s) with high confidence score(s) over communications network with modem; predictions may include the class name,
i.e. what the object is, a confidence percentage, and a list of coordinates detailing a bounding box of where the object is located/positioned in the image.).
At the time of effective filing, it would have been obvious to a person of ordinary skill in the art to selectively transmit information about the object in the method of CN114155299, Doumbouya et al., AOKI and Beard et al.
The suggestion/motivation for doing so would have been that to save resource.
Therefore, it would have been obvious to combine Langley et al. with Doumbouya et al., AOKI and Beard et al. and CN114155299 to obtain the invention as specified in claim 8.
With respect to claim 17, Langley et al. teach the edge communication engine is configured to transmit the compact message to the remote backend for less than all images frames (Fig. 5, col. 11 line 19- col. 14 line 24; flow chart for an exemplary method 500 for capturing and filtering images by the camera system 101; ref label 545, Filter image(s) using neural network(s) The neural network algorithms may be applied by the processor 230 in order to detect objects 120 within the images and to assign confidence scores to objects; ref label 550, Remove image(s) with low confidence score(s); ref label 555, Retain/store image(s) with high confidence score(s); ref label 565, Transmit retained image(s) with high confidence score(s) over communications network with modem).
Claims 18 is rejected under 35 USC 103 as being unpatentable over CN114155299 in view of Xia et al. (“Towards Semantic Integration of Machine Vision Systems to Aid Manufacturing Event Understanding “).
With respect to claim 18, CN114155299 teaches a first image sensor positioned in a physical space (page 7, 12th para., one of photographing devices) ;
a first positioning subsystem configured to determine first positional data for an object in a first image frame (IF) captured by the first image sensor; a first camera configured with a convolutional neural network (CNN) trained based on an object library configured to process the first IF for an object library associated with a digital twin, and configured to, for an object in the first IF using the object library and the first positional data, determine object information in the first IF including an object class, an object direction, a pose estimation, and object coordinates (page 8, 6th para., the YOLOv5 target detection algorithm may be used to perform the two-dimensional target detection step in step S102): (page 8, 5th para., the two-dimensional target detection result at least includes: the category to which each entity object belongs), (page 8, 13th para., the three-dimensional pose detection model is used to predict the entity position and entity direction of each entity object in the image frame to be processed in the first building scene. (page 6 2nd para. predict the entity position and entity direction of each of the entity objects in the to-be-processed image frame);
a second image sensor positioned in the physical space (page 7, 12th para., another one of photographing devices) ;
a second positioning system configured to determine second positional data for an object in a second image frame (IF) captured by the second image sensor;a second camera configured with a convolutional neural network (CNN) trained based on an object library configured to process the second IF for the object library, and configured to, for an object in the second IF using the object library and the second positional data, determine object information in the second IF including an object class, an object direction, a pose estimation, and object coordinates; (page 8, 6th para., the YOLOv5 target detection algorithm may be used to perform the two-dimensional target detection step in step S102): (page 8, 5th para., the two-dimensional target detection result at least includes: the category to which each entity object belongs), (page 8, 13th para., the three-dimensional pose detection model is used to predict the entity position and entity direction of each entity object in the image frame to be processed in the first building scene. (page 6 2nd para. predict the entity position and entity direction of each of the entity objects in the to-be-processed image frame);
a global server (page 7, 11th para)
CN114155299 does not teach receive, over a high bandwidth interface, the object information of the first camera and the second camera, aggregate the object information of the first camera and the second camera as aggregated object information, synchronize the aggregated object information as synchronize aggregated object information, and transmit the synchronized aggregated object information to the digital twin.
Xia et al. teach a global server (page 16, Fig. 14, control network )receive, the object information of the first camera and the second camera, aggregate the object information of the first camera and the second camera as aggregated object information, synchronize the aggregated object information as synchronize aggregated object information, (page 5, The synchronized results from multiple vision sources, such as inspection cameras, thermal cameras, and unmanned drones, are expected to aid the machine event-understandings along with the signals from conventional industrial sensors) and transmit the synchronized aggregated object information to the digital twin. (page 16, Fig. 14, synchronized states from control network to digital twin).
At the time of effective filing, it would have been obvious to a person of ordinary skill in the art to synchronize data from multiple vision sources to build digital twin in the method of CN114155299.
The suggestion/motivation for doing so would have been that to resolve occlusion and make precise scene understanding.
With respect to high bandwidth interface, it is obvious to use high enough bandwidth interface to handle data for digital twin.
Therefore, it would have been obvious to combine Xia et al. with CN114155299 to obtain the invention as specified in claim 18.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Randolph Chu whose telephone number is 571-270-1145. The examiner can normally be reached on Monday to Thursday from 7:30 am - 5 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached on (571) 272-7778.
The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/RANDOLPH I CHU/
Primary Examiner, Art Unit 2667
Prosecution Insights