DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1-2 are pending in this Application.
Priority
This Application is a continuation application of U.S. Non-Provisional application Ser. No. 17/984,996, filed on 10 Nov. 2022, which is a continuation application of U.S. Non-Provisional application Ser. No. 16/425,782, filed on 29 May 2019, which claims the benefit of U.S. Provisional Application No. 62/677,562, filed on 29 May 2018, each of which is incorporated in its entirety by this reference.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Instant Application 18/891,285
US Patent 12,127,076
A method comprising:
accessing a first protocol defining a first step, in a sequence of steps, for operation by a user within a particular site, the first protocol specifying a target location within the particular site and a target condition proximal the target location
in response to detecting occupation of a device falling proximal the target location specified in the first protocol:
at a first time, accessing a first image captured from an optical sensor arranged proximal the target location; and
calculating a first position of the device proximal the target location at a first positional resolution based on the first image;
in response to the first position of the device falling within a first threshold distance of a first equipment unit within a field of view of the optical sensor, extracting a first condition from the first image corresponding to the first equipment unit; and
in response to identifying a deviation between the first condition and the target condition specified in the first protocol,
flagging a current protocol record representing a current instance of the protocol by the user.
1. A method for assisting execution of manual protocols at production equipment comprising:
accessing a first protocol defining a first step, in a sequence of steps, for operation by a user within a particular site, the first protocol specifying a target location within the particular site and a target condition proximal the target location;
accessing a first set of identifiers from a first set of wireless access points accessible by a mobile device associated with the user at the particular site;
based on the first set of identifiers of the set of wireless access points, identifying a first location within the particular site occupied by the mobile device;
in response to the first location falling proximal the target location specified in the first protocol:
at a first time, accessing a first image captured from an optical sensor arranged proximal the target location;
extracting a first set of features from the first image;
calculating a first position of the mobile device proximal the target location at a first resolution based on positions of the first set of features relative a constellation of reference features representing the target location;
in response to the first position of the mobile device falling within a first threshold distance of a first equipment unit within a field of view of the optical sensor, extracting a first condition from the first image corresponding to a particular value presented at the first equipment unit; and
in response to identifying a deviation between the first condition and the target condition specified in the first protocol:
flagging a current protocol record representing a current instance of the protocol by the user; and
replaying a first segment of the current protocol record containing the first step in the protocol
Claim 1 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of U.S. Patent No. 12,127,076. Although the claims at issue are not identical, they are not patentably distinct from each other because the US patent claim 1 is narrower than claim 1 of the instant Application. It has been held in court that the generic patented invention has been anticipated by the species. See In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993).
Instant Application 18/891,285
US Patent 11,528,582
A method comprising:
accessing a first protocol defining a first step, in a sequence of steps, for operation by a user within a particular site, the first protocol specifying a target location within the particular site and a target condition proximal the target location
in response to detecting occupation of a device falling proximal the target location specified in the first protocol:
at a first time, accessing a first image captured from an optical sensor arranged proximal the target location; and
calculating a first position of the device proximal the target location at a first positional resolution based on the first image;
in response to the first position of the device falling within a first threshold distance of a first equipment unit within a field of view of the optical sensor, extracting a first condition from the first image corresponding to the first equipment unit; and
in response to identifying a deviation between the first condition and the target condition specified in the first protocol,
flagging a current protocol record representing a current instance of the protocol by the user.
See claim 1 above
1. A method for assisting execution of manual protocols at production equipment comprising:
accessing a geospatial location of a mobile device;
identifying a particular site occupied by the mobile device based on the geospatial location;
accessing identifiers of a first set of wireless access points wirelessly accessible to the mobile device;
based on identifiers of the set of wireless access points and known locations of wireless access points within the particular building, identifying a particular space within the particular building within the particular site occupied by the mobile device;
loading a protocol associated with an equipment unit in the particular space isolated and disconnected from local computer networks, the protocol defining a sequence of steps for operation of the equipment unit by a user;
detecting a first set of optical features in a field of view of an optical sensor integrated into the mobile device;
calculating a first position of the mobile device within the particular space at a first resolution in response to proximity of the mobile device to a low-risk zone defined in a space model of the particular space and associated with a first step in the protocol, and based on positions of the first set of optical features relative to a constellation of reference features represented in the space model of the particular space (B and C are suggested herein;
in response to the first position of the mobile device falling within a first threshold distance of a first reference location proximal the equipment unit associated with the first step of the protocol, serving a first guidance for the first step to the user, the first guidance comprising a prompt to capture data at a second reference location proximal the equipment unit following execution of a preceding control action at the first reference location proximal the equipment unit;
serving a prompt to cease motion for a period of time in response to proximity of the mobile device to a high-risk zone defined in the space model and associated with a second step in the protocol;
detecting a second set of optical features in the field of view of the optical sensor during the period of time;
calculating a second position of the mobile device in proximity to the high-risk zone within the particular space at a second resolution greater than the first resolution based on positions of the second set of optical features relative to the constellation of reference features represented in the space model of the particular space; and
in response to the second position of the mobile device falling within a second threshold distance of the second reference location proximal a particular display of the equipment unit associated with the second step of the protocol:
accessing a second image captured by the optical sensor within the second threshold distance of the second reference location of the equipment unit during the period of time;
isolating a region in the second image corresponding to the particular display of the equipment unit;
extracting a first value from the region in the second image corresponding to a particular value presented on the particular display; and
storing the first value in a current protocol record containing a set of values captured during a current instance of the protocol; and
at a remote computer system, in response to identifying a first deviation between the set of values of the current protocol record and a previous set of values of previous instances of the protocol, replaying a first segment of the current protocol record for the current instance of the protocol that spans the second step containing the first deviation.
20. A method for assisting execution of manual protocols at production equipment comprising:
at a mobile device during a first time period:
accessing a geospatial location of the mobile device;
identifying a particular site occupied by the mobile device based on the geospatial location;
accessing identifiers of a first set of wireless access points wirelessly accessible to the mobile device;
based on identifiers of the set of wireless access points and known locations of wireless access points within the particular building, identifying a particular space within the particular building within the particular site occupied by the mobile device;
loading a protocol associated with an equipment unit in the particular space disconnected from local area networks and local wireless networks, the protocol defining a sequence of steps for operation of the equipment unit by a user;
detecting a second set of wireless signals at the mobile device; and
calculating a first position of the mobile device at a first resolution in response to proximity of the mobile device to a low-risk zone defined in a space model of the particular space relative to the equipment unit based on positions of the second set of wireless signals and the constellation of wireless features represented in the space model;
at the mobile device during a second time following the first time:
serving a prompt to cease motion for a period of time in response to proximity of the mobile device to a high-risk zone defined in the space model and associated with a step in the protocol;
detecting a set of optical features in a first image captured by the optical sensor and depicting a field of view of the optical sensor during the period of time;
calculating a second position of the mobile device in proximity to the high-risk zone within the particular space at a second resolution greater than the first resolution based on positions of the set of optical features relative to the constellation of reference features represented in the space model of the particular space; and
in response to the second position of the mobile device falling within a second threshold distance of a second reference location proximal a particular display of the equipment unit associated with the step of the protocol:
isolating a region in the first image corresponding to the particular display of the equipment unit;
extracting a first value from the region in the second image corresponding to a particular value presented on the particular display; and
storing the first value in a current protocol record containing a set of values captured during a current instance of the protocol; and
at a remote computer system:
accessing a reference protocol record for the protocol;
detecting a deviation between the set of values in the current protocol record and a reference set of values in the reference protocol record;
in response to the deviation exceeding a deviation threshold, queuing a segment of the protocol record containing the deviation for remote verification; and
replaying the segment of the current protocol record that spans the step containing the first deviation.
Claim 1 is rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1 or 20, respectively of U.S. Patent No. 11,528,582 in view of Jun (KR 2017-0057854A as supported by the machine translation provided).
Claim 1 or 20 of the US patent 11528582 recites a method for assisting execution of manual protocols at production equipment. The Claim 1 or 20 of the US patent 11528582 teaches all the limitations of claim 1 of the current instant application but it does not explicitly teach flagging a current protocol record representing a current instance of the protocol by the user.
However, Jun in an analogous art of image capture and analysis, teaches a measuring equipment management system comprising a first image corresponding to component/display of an equipment unit (see Fig. 1 measurement equipment 100 including a camera for capturing images; see page 2 par. 7 “receiving an image of an analog instrument panel, And a control unit for extracting measurement information including a number, a scale, and a position of a display means for displaying the measurement value of the measurement object on the instrument panel, and extracting a measurement value of the analog instrument using the measurement information”, thus the image is from a panel and a region is isolated in order to read the values; also, see page 4 par. 10 “can use the camera 110 installed in the vicinity of the analogue instrumentation instrument 2 to generate a video image of the instrumentation board of the analogue instrumentation instrument 2 And analyze the collected images using the data processing device 130 to calculate the measurement values indicated by the measurement needles or gauges of the analog measurement equipment 2”); extracting a first condition from the first image corresponding to a particular value presented at the first equipment unit (see page 2 par. 7 “receiving an image of an analog instrument panel, And a control unit for extracting measurement information including a number, a scale, and a position of a display means for displaying the measurement value of the measurement object on the instrument panel, and extracting a measurement value of the analog instrument using the measurement information”); and o in response to identifying a deviation between the first condition and a target condition specified in a first protocol, flagging a current protocol record representing a current instance of a protocol by the user (see page 2 par. 7 “The controller may determine whether the measured value is within a normal range, and may transmit the measured value to a management server provided outside if the measured value is out of a normal range”, thus, when a deviation is detected the data is flagged by sending collected unusual data to the server; and see par. before last “determine whether the measurement value is within a normal range stored in the memory corresponding to the measurement data information”; see page 3 par. 2 “The image may include a color area displayed on the instrument panel, and the color area may be an area for displaying a caution or displaying a warning when the measured value falls within a certain range according to the position of the display part”, thus, the current data is flagged; also, see page 11 pars. 2-5 “The analog measuring equipment 2 may include a color area for displaying a warning or displaying a warning when the measurement result falls within a certain range depending on the position where the gauge needle corresponds to the instrument panel. Typically, the color area showing attention is yellow, and the color area indicating warning is red. The analog measuring equipment management system 100 according to the present embodiment detects a color area displaying an attention through color extraction from an image and a color area displaying a warning”, ). And see page 11 pars. 5 and 8 “…According to an embodiment of the present invention, it is possible to solve the problem of time and safety that can be generated in existing facilities, and to send a warning to a manager when the gauge is out of the normal operating range”).
Therefore, it would have been obvious to one of ordinary skilled in the art before effective filing date of the claimed invention to which said subject matter pertains to have modified the US patent 11528582’s invention as taught above to include the functions of the measuring equipment management system of Jun into the AR/VR data capture system of the US patent 11528582’s invention comprising taking a first image, extracting a first condition from the first image corresponding to a particular value presented at the first equipment unit and in response to identifying a deviation between the first condition and the target condition specified in the first protocol, flagging a current protocol record representing a current instance of the protocol by the user as taught by Jun in order to avoid human error in collecting and interpreting equipment data (Jun teaches a system that captures data, analyses and output second guidance such as alarm,/warnings, colors, automatically to indicate that there is a problem in the equipment), and prevent abnormal operation of the equipment and improve the service life and reliability of the equipment by providing guidance for a second step such as repair or maintenance (see page 11 par. 8 “According to an embodiment of the present invention, it is possible to solve the problem of time and safety that can be generated in existing facilities, and to send a warning to a manager when the gauge is out of the normal operating range, Repair. This makes it possible to prevent abnormal operation of the equipment and improve the service life and reliability of the equipment).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 2 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because Claim 2 recites “the inventions as shown and/or described herein” which does not recite at least one of the four categories of patent eligible subject matter recited in 35 USC 101 such as a process, machine, manufacture, or composition of matter.
Claims 1 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract without significantly more.
Claim 1 recites: “
accessing a first protocol defining a first step, in a sequence of steps, for operation by a user within a particular site, the first protocol specifying a target location within the particular site and a target condition proximal the target location;
in response to detecting occupation of a device falling proximal the target location specified in the first protocol:
at a first time, accessing a first image captured from an optical sensor arranged proximal the target location; and
calculating a first position of the device proximal the target location at a first positional resolution based on the first image;
in response to the first position of the device falling within a first threshold distance of a first equipment unit within a field of view of the optical sensor, extracting a first condition from the first image corresponding to the first equipment unit; and
in response to identifying a deviation between the first condition and the target condition specified in the first protocol, flagging a current protocol record representing a current instance of the protocol by the user”.
Under the broadest reasonable interpretation, the terms of the claim are presumed to have their plain meaning consistent with the specification as it would be interpreted by one of ordinary skill in the art. See MPEP 2111.
These limitations, as drafted, are a process that, under its broadest reasonable interpretation, covers steps of collecting information/observations (such as A) accessing a first protocol defining a first step, in a sequence of steps, for operation by a user within a particular site, B) in response to detecting occupation of a device falling proximal the target location specified in the first protocol: accessing a first image captured from an optical sensor arranged proximal the target location; and calculating a first position of the device proximal the target location at a first positional resolution based on the first image; C) extracting a first condition from the first image corresponding to the first equipment unit), Analyzing/evaluations/calculations steps (such as B) in response to detecting occupation of a device falling proximal the target location specified in the first protocol, calculating a first position of the device proximal the target location at a first positional resolution based on the first image; C) in response to the first position of the device falling within a first threshold distance of a first equipment unit within a field of view of the optical sensor; and performing judgements and opinions (such as D) in response to identifying a deviation between the first condition and the target condition specified in the first protocol, flagging a current protocol record representing a current instance of the protocol by the user), and which are steps that can be easily performed mentally. For instance, these steps were conventional steps that were performed manually and visually by a user/technician whenever they had to collect data from a display of a machine/equipment in a plant located within a building and consisted on retrieving protocols/manuals establishing location and expected target values of the collected values, determine the distances to the equipment, calculate deviations of the values collected and . A user/technician goes to a place, access or reads the steps or tasks from a paper guide to be performed at certain location and read a target condition at the location, reads data from access points for identifying the location of floors or areas in order to reach his destination, access images mentally and with his eyes and extract images to determine the best pose to collect data, then when the user/technician reach the target location proceeds to perform the steps/tasks such as mentally collect data from displays/meters/gauges and mentally generates a judgement/flag or alarm that there is problem with the current protocol tasks such when a value read does not coincide with an expected known value. Thus, the claim recites a mental process.
The judicial exceptions are not integrated into a practical application because the claims recite the additional elements such as, accessing a first image captured from an optical sensor (capturing an image with a mobile device camera), extracting a first condition from the first image corresponding to the first equipment unit and which are recited in a high level of generality and generally links the judicial exception to a technological environment of using computer (mobile devices and cameras) for the purposes of localization and image capture of data (see MPEP 2106.05(h) ) and/or are insignificant extra solution activity which are well understood (see MPEP 2106.05(g) and 2106.05(d)).
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of using a device for accessing a first image captured from an optical sensor (capturing an image with a mobile device camera), extracting a first condition from the first image corresponding to the first equipment unit, which are recited in high level of generality and simply represents no more than instructions “to apply” with the judicial exception/the abstract idea on a computer or to generally link the use of the judicial exception to the technological environment of a mobile computer cannot provide an inventive concept as stated by the courts (see MPEP 2106.05(f) and MPEP 2106.05(h)). For instance, using mobile devices and cameras for gathering data for localization purposes and for image data collection were well known and ‘well-understood, routine, conventional activities” previously known to the industry (see MPEP 2106.05(f)). For instance, the reference Santarone et al (US 20180239840) teaches or suggests using a mobile device identifying a first location within the particular site based on the first set of identifiers of the set of wireless access points (see [0065] and [0286]), and an optical sensor/camera for accessing a first image captured from an optical sensor arranged proximal the target location: o extracting a first set of features from the first image; o calculating a first position of the mobile device proximal the target location at a first resolution based on positions of the first set of features relative a reference features representing the target location (see [0073] and [0198], see the rejection below). Reiley et al (US 20180357907), also teaches an optical sensor/camera for accessing a first image captured from an optical sensor arranged proximal the target location: o extracting a first set of features from the first image; o calculating a first position of the mobile device proximal the target location at a first resolution based on positions of the first set of features relative a reference features representing the target location (see 0019 and see [0017], [0019], [0032], [0034], [0040], [0043], and [0046]). Also, Humberto Sossa et al (Automatic Reading of Electro-mechanical Utility Meters), Robert Sablatnig et al (Application Constraints in the Design of an Automatic Reading Device for Analog Display Instruments), Depari et al (Using smartglasses for utility-meter reading), respectively teach using an optical sensor to capture data of a display connected to an equipment in order to detect a condition or value in the image (see the Figs. In each reference).
Accordingly, these additional elements do not integrate the abstract idea into a practical application, do not amount to significantly more than the judicial exception, and do not impose any meaningful limits on practicing the abstract idea. Therefore, the claims are not patent eligible.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 2 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 2 recites “the inventions as shown and/or described herein”. These limitations are unclear and confusing because it is unclear to what “the inventions” are considered. The term “the inventions” lack of antecedent basis. Thus, it is unclear to what is referring to.
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claim 2 is rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends.
Claim 2 recites “the inventions as shown and/or described herein”. However, claim 2 is not dependent and/or further limiting claim 1. Thus, claim 2 does not further limit claim 1. Even if claim 2 was dependent on claim 1, it would not be further properly limiting claim 1 as recited because it does not recites additional limitation that further limit claim 1.
Applicant may cancel the claim(s), amend the claim(s) to place the claim(s) in proper dependent form, rewrite the claim(s) in independent form, or present a sufficient showing that the dependent claim(s) complies with the statutory requirements.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 is rejected under 35 U.S.C. 103 as being unpatentable over Santarone et al (US 20180239840) in view of Jun (KR 2017-0057854A as supported by the machine translation provided).
As per claim 1, Santarone teaches a method (see protocols have been interpreted in BRI has tasks, steps, guidance as suggested in the disclosure; see [0008-009] method and apparatus for assisting in the collection of data) comprising:
accessing a first protocol defining a first step (see protocols have been interpreted in BRI has tasks, steps, guidance as suggested in the disclosure; see [0008-009] method and apparatus for assisting in the collection of data; also, see [0146] the AR/VR system is for assisting a user and present him with data including experiential data (includes collection of data), maintenance data, and annotations (include operations instructions, 0147); 0058 “In another aspect, an Augmented Virtual Model may be accessed in relation to modelling achievement of a stated Performance Level. Accurate capture of As Built Features and aggregated data of similar buildings, equipment types, machinery and usage profiles assist in one or more of:…providing on site guidance during the Service Call; providing documentation relevant to the building, equipment and machinery; providing access to remote experts that guide onsite technicians”; also, see [0259] “…indicate completion of a task”), in a sequence of steps, for operation by a user within a particular site, the first protocol specifying a target location within the particular site and a target condition proximal the target location (see the terms target condition and target location are not explicitly recited in the disclosure, and will be interpreted in the BRI in light of the specification. For instance, the target location is a distance threshold that causes activation of a model and/or protocol. The target location also could be the model associated with of the building or areas where the equipment is located, wherein the models of the building or areas are activated when the user/mobile device is detected near the areas. The target condition in the BRI in light of the disclosure is a reference value or threshold of a measured/detected value which is compared to a current detected value .
Santarone in Figs. 1E, 3H, 4A and 4C steps 408-409 suggests that when a user is detected within a specific target location, the system access a protocol defining a first step of downloading an augmented virtual model (AVM) and presenting steps or guidance to a user. The protocol specifies a target location which is a distance threshold that causes the accessing/downloading of the AVM. Santarone further teaches [0065] “Some exemplary embodiments may include updates to an AVM that include, one or more of: quantifying a make and model of equipment and machinery on site; time and date notation of change in location specific data; Model accessed and/or updated according to XYZ and distance data; XY data may include high level location designation within the street address via triangulation (i.e. such as a street address) and highly specific position designation (i.e. particular room and wall)”; [0120]”… a threshold of KW hours of electricity consumed by the structure on a monthly basis…”; [0195] “…data to be acquired, relating to life expectancy of equipment, may include, but is not limited to, hours of operation, conditions of operation (whether and how long the equipment may be running under capacity, at rated capacity, or over capacity), or many environmental conditions for operation; environmental conditions may include the ambient temperature (or the difference in ambient temperature from an ideal or other measured value…”; also, [0286] “At method step 408, an AVM may be identified and accessed via the physical location. Once an appropriate AVM is accessed, a particular portion of the AVM may be presented via a GUI based upon the position within the Processing Facility (or proximate to the Processing Facility) and a direction, height and angle of view. The position may be determined relative to location identifiers. Height may be determined via electronic devices, such as a smart device, or via triangulation referencing the location identifiers (locations
identifiers are discussed more fully above and below)”, Thus, the protocol specifies a target location; also, see [0291] “…e) calculating a position within the Processing Facility, the calculation based upon a relative distance of the at least three positioning reference devices to the point of measurement and a triangulation calculation…”; also, see [0297-0298]);
in response to detecting occupation of a device falling proximal the target location specified in the first protocol (see page 27 claim 19 “generating a volume of production of a machine located at the Z distance of interest and the Z direction of interest from the XY position of the smart device”, thus, data of the AVM is generated when the user/mobile device is within a specific distance; also, see [0065] “Model accessed and/or updated according to XYZ and distance data”; also, see [0141] “Based upon a recorded position within the structure 102A and the calculated direction, a data record may be accessed in the Augmented Virtual Model 100 and a specific portion of the Augmented Virtual Model 100 and/or the Virtual Processing Facility 102B may be presented on the user device 106”, again, in response to a position within a specific distance threshold of the mobile device with respect to equipment, the AVM and all the guidance/protocol is allowed to take place at the position. Also, see [0154] “The user 131 may also be provided with controls 133. Controls 133 may include, for example, zoom, volume, scroll of data fields and selection of data fields. Controls may be operated based upon an item of Equipment 132 within a Field of View 138 of the User 131 located at a vantage point 137 and viewing a selected direction (Z axis).Also, see [0286]-[0287] and [0316] and Fig. 9A-D and Figs. 10-12”; see Fig. 1E the first step could include a directional audio capture):
at a first time, accessing a first image captured from an optical sensor arranged proximal the target location (see [0198] “include image based identification; a device with some imaging means, including but not limited to a mobile device camera, tablet device camera, computer camera, security camera, or AR headset camera may image the equipment to be identified. Image recognition software may be used to identify the visualized equipment by its identifying features”; also, see [0075] “Image Capture Device, such as a Charge Coupled Device ("CCD") capture device and/or an infrared capture device being available in a handheld [0078] “captured data may be compared to a library of stored data using image recognition software to ascertain and/or affirm a specific location, elevation and direction of an image capture location”; also, see [0078] and [0234]; also, see Fig. 3G, 3H, the mobile device has a camera for imaging and thus detecting optical features in a field of view; also, see [0154]); and
calculating a first position of the device proximal the target location at a first positional resolution based on the first image (see [0073] “calculate a transform that projects the known features in the geospatial cluster onto corresponding image features in the optical image (or video frame, etc.); and apply the transform to geographic locations associated with these known features to calculate an actual 3D geographic location of the mobile computing device; and see [0075] “Image Capture Device, such as a Charge Coupled Device ("CCD") capture device and/or an infrared capture device being available in a handheld [0078] “captured data may be compared to a library of stored data using image recognition software to ascertain and/or affirm a specific location, elevation and direction of an image capture location”; also, see [0297-0298] “Other embodiments may include a device recognizable via image analysis and a camera or other Image Capture Device, such as a CCD device, may capture an image of three or more position identifiers 501-504. Image analysis may recognize the identification of each of three or more of the position identifiers 501-504 and a size ratio of the respective image captured position identifiers 501-504 may be utilized to calculate a precise position. Similarly, a height designation may be made via triangulation using the position identifiers as reference to a known height or a reference height”);
in response to the first position of the device falling within a first threshold distance of a first equipment unit within a field of view of the optical sensor (see Fig. 1E capture of directional image data, 3H, 5, 9A-D and 12l; also, see [0098] “Directional Image Data” as used herein refers to image data captured from a Vantage Point…”; and see [0198]), extracting a first condition from the first image corresponding to the first equipment unit (see page 27 claim 19 “generating a volume of production of a machine located at the Z distance of interest and the Z direction of interest from the XY position of the smart device”, thus, data of the AVM is generated when he user/mobile device is within a specific distance threshold based on the image analysis; also, see [0065] “Model accessed and/or updated according to XYZ and distance data”; also, see [0141] “Based upon a recorded position within the structure 102A and the calculated direction, a data record may be accessed in the Augmented Virtual Model 100 and a specific portion of the Augmented Virtual Model 100 and/or the Virtual Processing Facility 102B may be presented on the user device 106”, again, in response to a position within a specific distance threshold of the mobile device with respect to equipment, the AVM and all the guidance/protocol is allowed to take place at the position. Also, see [0286]-[0287] and [0316] and Fig. 9A-D and Figs. 10-12”; see Fig. 1E the first step could include a directional audio capture/condition; also, see page 26 claim 1 “performing a data capture routine from the XY position, said data capture routine generating a digital record of a condition of the commercial structure noted at the Z distance of interest and the Z direction of interest from the XY position of the smart device and the elevation of the smart device…and generating a virtual representation of the digital record of a condition of the commercial structure within the Augmented Virtual Model”; see page 27 “claim 10 “performing a data capture routine from the XY position, said data capture routine generating a digital record of a condition of the commercial structure noted at the Z distance of interest and the Z direction of interest from the XY position of the smart device and the elevation of the smart device; transmit the digital record of a condition of the commercial structure noted at the Z distance of interest from the XY position of the smart device to the Augmented Virtual Model; and generating a virtual representation of the digital record of a condition of the commercial structure within the Augmented Virtual Model also, see 0078 “In another aspect, captured data may be compared to a library of stored data using image recognition software to ascertain and/or affirm a specific location, elevation and direction of an image capture location and proper alignment with the virtual model. Still other aspects may include the use of a compass incorporated into a Smart Device”; also, see [0119]; also, see 0081 “…on site data capture may include designation of an XYZ reference position and one or more of: image capture; infra-red capture; Temperature; Humidity; Airflow; Pressure/tension…”; also, see [0195] “…data to be acquired, relating to life expectancy of equipment, may include, but is not limited to, hours of operation, conditions of operation (whether and how long the equipment may be running under capacity, at rated capacity, or over capacity), or many environmental conditions for operation; environmental conditions may include the ambient temperature (or the difference in ambient temperature from an ideal or other measured value), ambient humidity (or the difference in ambient humidity from an ideal or other measured value),…”);
While Santarone teaches capturing images and detecting the location of user/mobile device based on image analysis, and while Santarone clearly teaches extracting condition/data capture from or of the facility/equipment, Santarone does not explicitly teach in response to identifying a deviation between the first condition and the target condition specified in the first protocol, flagging a current protocol record representing a current instance of the protocol by the user.
However, Jun in an analogous art of image capture and analysis, teaches a measuring equipment management system comprising a first image corresponding to component/display of an equipment unit (see Fig. 1 measurement equipment 100 including a camera for capturing images; see page 2 par. 7 “receiving an image of an analog instrument panel, And a control unit for extracting measurement information including a number, a scale, and a position of a display means for displaying the measurement value of the measurement object on the instrument panel, and extracting a measurement value of the analog instrument using the measurement information”, thus the image is from a panel and a region is isolated in order to read the values; also, see page 4 par. 10 “can use the camera 110 installed in the vicinity of the analogue instrumentation instrument 2 to generate a video image of the instrumentation board of the analogue instrumentation instrument 2 And analyze the collected images using the data processing device 130 to calculate the measurement values indicated by the measurement needles or gauges of the analog measurement equipment 2”); extracting a first condition from the first image corresponding to a particular value presented at the first equipment unit (see page 2 par. 7 “receiving an image of an analog instrument panel, And a control unit for extracting measurement information including a number, a scale, and a position of a display means for displaying the measurement value of the measurement object on the instrument panel, and extracting a measurement value of the analog instrument using the measurement information”); and o in response to identifying a deviation between the first condition and a target condition specified in a first protocol, flagging a current protocol record representing a current instance of a protocol by the user (see page 2 par. 7 “The controller may determine whether the measured value is within a normal range, and may transmit the measured value to a management server provided outside if the measured value is out of a normal range”, thus, when a deviation is detected the data is flagged by sending collected unusual data to the server; and see par. before last “determine whether the measurement value is within a normal range stored in the memory corresponding to the measurement data information”; see page 3 par. 2 “The image may include a color area displayed on the instrument panel, and the color area may be an area for displaying a caution or displaying a warning when the measured value falls within a certain range according to the position of the display part”, thus, the current data is flagged; also, see page 11 pars. 2-5 “The analog measuring equipment 2 may include a color area for displaying a warning or displaying a warning when the measurement result falls within a certain range depending on the position where the gauge needle corresponds to the instrument panel. Typically, the color area showing attention is yellow, and the color area indicating warning is red. The analog measuring equipment management system 100 according to the present embodiment detects a color area displaying an attention through color extraction from an image and a color area displaying a warning”, ). And see page 11 pars. 5 and 8 “…According to an embodiment of the present invention, it is possible to solve the problem of time and safety that can be generated in existing facilities, and to send a warning to a manager when the gauge is out of the normal operating range”).
Therefore, it would have been obvious to one of ordinary skilled in the art before effective filing date of the claimed invention to which said subject matter pertains to have modified Santarone’s invention as taught above to include the functions of the measuring equipment management system of Jun into the AR/VR data capture system of Santarone’s invention comprising taking a first image, extracting a first condition from the first image corresponding to a particular value presented at the first equipment unit and in response to identifying a deviation between the first condition and the target condition specified in the first protocol, flagging a current protocol record representing a current instance of the protocol by the user as taught by Jun in order to avoid human error in collecting and interpreting equipment data (Jun teaches a system that captures data, analyses and output second guidance such as alarm,/warnings, colors, automatically to indicate that there is a problem in the equipment), and prevent abnormal operation of the equipment and improve the service life and reliability of the equipment by providing guidance for a second step such as repair or maintenance (see page 11 par. 8 “According to an embodiment of the present invention, it is possible to solve the problem of time and safety that can be generated in existing facilities, and to send a warning to a manager when the gauge is out of the normal operating range, Repair. This makes it possible to prevent abnormal operation of the equipment and improve the service life and reliability of the equipment).
Conclusion
The prior art made of record and not relied upon, as cited in PTO form 892, is considered pertinent to applicant's disclosure.
Wang (US 8341741) teaches a system for capturing data with a camera of a display of an equipment (see Col 2 lines 6-12 “The image or video collecting device is preferably a camera for shooting data video image shown on the display of the process control computer, or a video collecting device for acquiring video signal from the display of the process control computer);
Meganathan et al (US 20150296188), teaches a system comprising determining the location of a person in a low risk and high risk areas (see 0046);
Schmirler et al (US 20180130260) teaches an VR and AR system comprising presenting protocols to a user via a mobile device, detecting the location of the mobile device in an industrial plant (see Fig. 7), collecting data of the industrial plant based on the location of the mobile device (see Fig. 8).
Russel (10354137) teaches a system comprising imaging a dial, when a detected value extracted from the image has a deviation, and alert is transmitted (see Fig. 2 and 3).
Glaser (US 10615994) a mobile device to capture data of a dial, isolate a region in the image corresponding to a display, and extract data (see Fig. 2 and Fig. 36) and determine if there is a deviation to trigger an alarm, suggests adjustments when there are capture issues.
Leigh (US 10878607) teaches a mobile inspection device (104) to capture data of an equipment including a display/dial, isolate a region in the image corresponding to a display (see fig. 2), and extracting a first value from the region in the second image corresponding to a particular value presented on the particular display; and in response to the first value falling within a value range threshold associated with the first step of the procedure, may display an alert (e.g., via inspection device 104) to operator 106 indicating, for example, an error in the sample image (e.g., perhaps operator 106 captured the sample image from a similar but incorrect process indicator), or an asset operating outside of norms for that asset or store the value captured.
Mullins et al (US 9412205) teaches a mobile inspection device (104) to capture data of an equipment including a display/dial, isolate a region in the image corresponding to a display, and extracting a first value from the region in the second image corresponding to a particular value presented on the particular display; and in response to the first value falling within a value range threshold associated with the first step of the procedure, may display an alert a red color (see Figs. 12A-FB).
Harmon (US 11093777) and Chrichlow (US 20060036967), respectively, teach a camera imaging an area, isolating the area of a display, extracting data from the display, but does not teach in response to the first value falling within a value range threshold associated with the first step of the procedure, serving a second guidance for a second step of the procedure.
Chattopadhyay (US 20170089692) teaches a camera imaging an area, isolating the area of a display, extracting data from the display, and in response to the first value falling within a value range threshold associated with the first step of the procedure display an alert/flag (see Fig. 2).
Barros et al (US 20020147717), teaches a system comprising detecting the movement and current location of a mobile device and send data according to the location (see Abstract, ) detecting the user/mobile device in different zones (abstract).
Reiley et al (US 20180357907) teaches a system for dispatching a vehicle comprising taking a first image, detect and extract features form the first image, and detect a location at a first resolution based on the image, and taking one or more images to detect the location of the device.
Miller et al (US 8606544), Margalit (US 20110133940), Almadi et al (US 20140114598), Pressman et al (US 11327121) teaches a system flagging a system when a measurements value deviates from a target value in order to indicate the deviation (see page 30 Col 2 par. 4). Thus, flagging a system during execution of a protocol/algorithm when monitoring equipment based on deviation of measured values from a target values was very well known.
Gao (EP 3174319 B1), teaches a system and method for location services comprising detecting the location of a device by using Wi-Fi networks, routers, access pints (see 0056, 0147).
Villarubia et al (Hybrid Indoor Location System for Museum Tourist
Routes in Augmented Reality) teaches a system comprising for indoor localization of user mobile device based on WIFI signals and then refining the location based images obtained from a camera.
Haramaki et al, (A Network Topology Visualization System Based on Mobile AR Technology, 2015) teaches a system a localization system comprising using wireless signal strength for location purposes and also using images to further estimate the location of a user. The system provides an augmented reality (AR) image based on the location of the device and based on the image detected and provides information that assist a user to perform setting tasks.
Duy et al (A Survey of Fingerprint-Based Outdoor Localization, 2016) teaches a system for localization of a device based on fingerprints of image detected, fingerprints of signal strength of AP devices. However, none of the references above alone or in combination teach the combination of limitations of claim 1 as claimed.
Examiner respectfully requests, in response to this Office action, support be shown for language added to any original claims on amendment and any new claims. That is, indicate support for newly added claim language by specifically pointing to page(s) and line number(s) in the specification and/or drawing figure(s). This will assist Examiner in prosecuting the application.
When responding to this Office Action, Applicant is advised to clearly point out the patentable novelty which he or she thinks the claims present, in view of the state of the art disclosed by the references cited or the objections made. Applicant must also show how the amendments avoid or differentiate from such references or objections. See 37 CFR 1.111 (c).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to OLVIN LOPEZ ALVAREZ whose telephone number is (571) 270-7686 and fax (571) 270-8686. The examiner can normally be reached Monday thru Friday from 9:00 A.M. to 6:00 P.M.
If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Robert Fennema, can be reached at (571) 272-2748. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center for authorized users only. Should you have questions about access to Patent Center, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/O. L./
Examiner, Art Unit 2117
/DARRIN D DUNN/Patent Examiner, Art Unit 2117
Prosecution Insights