Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation is: “a visual renderer” in claim 9.
Because this claim limitation is being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it is being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this limitation interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation recites sufficient structure to perform the claimed function so as to avoid it being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
For the sake of further prosecution, the Examiner will treat the respective “visual renderer” as appropriate hardware configured to perform the recited functions.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 6-8, 10, 13-15, 17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Gruteser (US20210110191A1) and Baldwin (US20230231984A1).
Regarding claim 8, Gruteser teaches a system for optimizing edge-assisted augmented reality (AR) devices, comprising: one or more AR devices and an edge server (Gruteser; ¶0007, describes a system that operates on a mobile device, such as an AR device, and dynamically offloads computationally intensive functions to an edge cloud device.) analyze requests from the AR devices (Gruteser; ¶0090, describes the server side receives encoded frame slices (frame processing requests) from the AR device, decodes and inferences, and sends detection results and a QP map back to the AR device. This teaches analyzing requests of the AR device.) and further teaches (Gruteser; ¶0090) the edge cloud sends detection results and a QP map (response packet) back to the AR device for future processing and (Gruteser; ¶0076) the pipeline sends the QP map (response packet) back to the AR device, which uses it for the next captured frame.
However, Gruteser does not explicitly disclose the memory device operatively coupled with one or more processors, profiling frame capture timings that capture relationships between AR devices, determining accuracy of frame capture timings based on a timing-accuracy SLO metric, or determining and applying camera frame timing adjustments based on synchronization relationships.
Baldwin teaches an edge server including a memory device operatively coupled with one or more processor devices (Baldwin; ¶0005, describes an apparatus that includes a memory coupled to one or more processors. In the combination, Baldwin’s processor and memory apparatus is implemented with Gruteser’s edge cloud device. This teaches the edge server including a memory device operatively coupled with one or more processor devices.), profile frame capture timings of the AR devices that capture relationships between the AR devices (Baldwin; ¶0041, describes camera synchronization software that samples the timing of each camera being synchronized, calculates synchronization error between the cameras , adjusts timing of the cameras to remove the synchronization error, waits for the timing update to apply, and then re-samples the timing of each camera. The sampled timing of each camera correlates to the profiled frame capture timings and the synchronization error identifies the relationship between the camera timings. In the combination, the cameras are cameras of the AR devices. This teaches profiling frame capture timings of the AR devices that capture relationships between the AR devices.) determine accuracy of the frame capture timings of the AR devices based on a service level objective (SLO) metric (Baldwin; ¶0069, describes that middle of exposure alignment can support multiple cameras running independent AE and maintain synchronization to 0.1 milliseconds or better and synchronization quality may be approximately one “sensor-line-time” (sensor timing granularity). The synchronization quality is a timing accuracy objective for the camera frame capture timings. This teaches determining accuracy of the frame capture timings based on a SLO metric.) determine a frame timing plan that minimizes overall timing changes of the AR devices (Baldwin; ¶0068, describes correcting the planned duration for frame N to force the synchronization error to 0, where the synchronization master (CamA) has no change and the other cameras have their frame N durations adjust by their respective synchronization errors. The planned duration for frame N correlates to the frame timing plan. The adjustment minimizes overall timing changes because the synchronization master is unchanged and the other cameras are adjusted by their respective synchronization errors. This teaches determining a frame timing plan that minimizes overall timing changes of the AR device.)
by adapting the accuracy of the frame capture timings to optimal adjustments generated based on a change in device metrics for requests below an accuracy threshold (Baldwin; ¶0065, describes state data including camera sensor read delay, start of frame time for the last captured frame, durations for pending frames and the frame being requested, and the exposure for the frame being requested. State data, duration of the frames, and exposure are camera metrics used to generate timing adjustments for the requested frame. As discussed above, Baldwin; ¶0068, teaches the planned duration correction teaches optimizing by making adjustments that use frame capture timing accuracy (forcing synchronization error to 0) and, Baldwin; ¶0069, teaches that requests below an accuracy threshold (synchronization error of 0.1 milliseconds or better) do not satisfy the timing accuracy objective. This teaches adapting the accuracy of the frame capture timings to optimal adjustments generated based on a change in device metrics for requests below an accuracy threshold.)
and adjust current frame capture timings of cameras of the AR devices based on the frame timing plan by generating a response packet for the AR devices (Baldwin; ¶0073 describes adjusting, based on the respective synchronization error, a duration of the frame (frame capture timing) at one or more cameras, where the adjusted duration aligns the common point at each camera for the frame. Adjusting the duration of the frame correlates to adjusting the camera frame capture timing and, as described above Baldwin; ¶0068, describes correcting the planned duration for frame N (frame timing plan). As previously discussed, Gruteser; ¶0090, detection results and QP map (response packet) are sent back to the AR device and, Gruteser; ¶0076, the AR device uses this for the next captured frame. In the combination, the edge server adjusts the current frame capture timings according to the frame timing plan of Baldwin by generating the response packet, which provides the adjustment, to the AR devices to apply. This teaches adjusting current frame capture timings of cameras of the AR devices based on the frame timing plan by generating a response packet for the AR devices.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the edge assisted AR frame processing system of Gruteser with the camera synchronization timing of Baldwin. The motivation for such a combination would have been to provide the benefit of improved frame timing accuracy.
Claim 1, has similar limitations as of claim 8, therefore it is rejected under the same rationale as claim 8.
Claim 15, has similar limitations as of claim 8, therefore it is rejected under the same rationale as claim 8, except claim 15 recites A non-transitory computer program product comprising a computer-readable storage medium including program code. Baldwin states “the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.”
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the edge assisted AR frame processing system of Gruteser as a computer program product stored on memory as taught by Baldwin. The motivation for such a combination would have been to provide the benefit of improved compatibility with modern computers and/or servers.
Regarding claim 10, Gruteser in view of Baldwin teaches the system of claim 8, wherein to profile the frame timings further comprises to assess a drop rate of the requests based on efficiency scores of offline frame timing capture plans (Baldwin; ¶0064, describes that frame request delay can be reduced by queueing fewer frames, but may risk frame drops and/or loss of camera synchronization; and further describes that a feed-forward camera synchronization algorithm can predict frame timestamps and synchronization errors in advance. The predicted frame timestamps and synchronization errors read on offline frame timing capture plans because they are calculated in advance for requested frames. The reduced frame request delay and predicted synchronization error correlate to efficiency scores for those plans and the risk of frame drops correlates to assessing a drop rate of the requests. This teaches assessing a drop rate of the requests based on efficiency scores of offline frame timing capture plans.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the frame-timing profiling of Gruteser to asses request drop rate based on efficiency scores of offline frame timing capture plans as taught by Baldwin. The motivation for such a combination would have been to provide the benefit of reducing dropped frame and synchronization failures.
Claim 2, has similar limitations as of claim 10, therefore it is rejected under the same rationale as claim 10.
Claim 16, has similar limitations as of claim 10, therefore it is rejected under the same rationale as claim 10.
Regarding claim 13, Gruteser in view of Baldwin teaches the system of claim 8, wherein to determine the frame timing plan further comprises fine-tuning the frame timing plans for the AR devices based on minimum and maximum timing change limits (Baldwin; ¶0069, describes maintaining a synchronization to 0.1 milliseconds. Synchronization quality may be approximately one “sensor-line-time” and that sensor timing granularity can vary. A camera sensor running at 90 Hertz (Hz) at resolution of 640×480 has a line-time of roughly 0.02 ms. The “sensor-line-time” granularity correlates to a minimum timing change limit and the 0.1 ms synchronization quality correlates to an upper timing error limit. This teaches fine-tuning the frame timing plans based on minimum and maximum timing change limits.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the frame-timing profiling of Gruteser to asses request drop rate based on efficiency scores of offline frame timing capture plans as taught by Baldwin. The motivation for such a combination would have been to provide the benefit of reducing dropped frame and synchronization failures.
Claim 6, has similar limitations as of claim 13, therefore it is rejected under the same rationale as claim 13.
Claim 20, has similar limitations as of claim 13, therefore it is rejected under the same rationale as claim 13.
Regarding claim 14, Gruteser in view of Baldwin teaches the system of claim 8, wherein to adjust the current frame capture timings further comprises adjusting the current frame capture timings gradually by 1 millisecond batches (As previously discussed in claim 8, Baldwin; ¶0041, describes camera synchronization software that samples the timing of each camera being synchronized, calculates synchronization error between the cameras , adjusts timing of the cameras to remove the synchronization error, waits for the timing update to apply, and then re-samples the timing of each camera. Baldwin; ¶0069, further describes that camera timing may be adjusted using sensor timing granularity (line-time of roughly 0.02 ms). A 1 millisecond batch is larger than Baldwin’s sensor timing granularity and, in the combination with Gruteser’s 60 Hz AR cameras, relatively small and allows for gradual correction, therefore is a predictable timing adjustment increment.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the frame-timing system of Gruteser with the batch sizes as taught by Baldwin. Adjusting the frame capture timings gradually by 1 millisecond batches applies known batch-timing-adjustment techniques to yield the predictable result of smooth timing correction within the sensor timings and frame rate as taught by Gruteser in view of Baldwin.
Claim 7, has similar limitations as of claim 14, therefore it is rejected under the same rationale as claim 14.
Claims 4-5, 11-12, and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Gruteser (US20210110191A1), Baldwin (US20230231984A1), and Martello (Martello, Silvano, and Paolo Toth. "Linear assignment problems." North-Holland Mathematics Studies. Vol. 132. North-Holland, 1987. 259-282.)
Regarding claim 11, Gruteser in view of Baldwin teaches the system of claim 8, including determining a frame timing plan that minimizes overall timing changes (Baldwin; ¶0068, see claim 8).
However, Gruteser in view of Baldwin does not explicitly disclose wherein to determine the frame timing plan further comprises transforming a generated cost matrix into a linear assignment problem to generate transition plans that minimizes overall timing changes.
Martello teaches transforming a generated cost matrix into a linear assignment problem to generate transition plans that minimizes overall timing changes (Martello; p. 260-261, section 2.1, describes that, given a cost matrix, the Min-Sum assignment problem finds a permutation that minimizes the sum of assigned costs, where each assignment has a cost and the assignment problem minimizes the total cost. In the combination, the timing transitions from each camera’s current planned duration to its corrected frame N duration (Baldwin; ¶0068, see claim 8) are the assignments and the timing changes for each transition are assignment costs in the (generated) cost matrix. Applying the linear assignment problem to the assignments generates transition plans that minimize total timing change cost. This teaches transforming a generated cost matrix into a linear assignment problem to generate transition plans that minimize overall timing changes.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the frame-timing plant of Gruteser in view of Baldwin with the cost-matrix optimization of Martello as this is a known optimization technique for minimizing cost to provide the benefit of minimizing timing changes.
Claim 4, has similar limitations as of claim 11, therefore it is rejected under the same rationale as claim 11.
Claim 18, has similar limitations as of claim 11, therefore it is rejected under the same rationale as claim 11.
Regarding claim 12, Gruteser in view of Baldwin teaches the system of claim 8,
including determining a frame timing plan that minimizes overall timing changes (Baldwin; ¶0068, see claim 8).
However, Gruteser in view of Baldwin does not explicitly disclose wherein to determine the frame timing plan further comprises generating optimal adjustments for the AR devices by seeking a minimum sum of costs by selecting a single element from each row, while adhering to a constraint that chosen elements must belong to distinct columns.
Martello teaches generating optimal adjustments for the AR devices by seeking a minimum sum of costs by selecting a single element from each row, while adhering to a constraint that chosen elements must belong to distinct columns (Martello; p. 260-261, section 2.1, describes the Min-Sum assignment problem using a cost matrix and minimizing the total assignment cost, with one item selected for each row and column used once. The problem selects one cost entry from each row and column. The timing transitions from each camera’s current planned duration to its correct frame N duration (Baldwin; ¶0068, see claim 8) are assignment choices and the timing change for each transition Is the assignment cost. This teaches generating optimal adjustments by selecting one timing transition choice of each device (distinct columns) and minimizing the total timing change cost.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the frame-timing plant of Gruteser in view of Baldwin with the cost-matrix optimization of Martello as this is a known optimization technique for minimizing cost to provide the benefit of minimizing timing changes.
Claim 5, has similar limitations as of claim 12, therefore it is rejected under the same rationale as claim 12.
Claim 19, has similar limitations as of claim 12, therefore it is rejected under the same rationale as claim 12.
Claims 2, 9, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Gruteser (US20210110191A1), Baldwin (US20230231984A1), and Sreejith (US20230021676A1)
Regarding claim 9, Gruteser in view of Baldwin teaches the system of claim 8, further comprising a visual renderer to render superimposed images of a detected object to the AR devices (Gruteser; ¶0064, describes a local rendering pipeline (visual renderer) that tracks the scene and renders virtual overlays (superimposed images) while waiting for object detection results and, Gruteser; ¶0089, describes rendering virtual overlays (superimposed images) based on coordinates of the detection result. This teaches rendering superimposed images of a detected object to the AR devices.)
However, Gruteser in view of Baldwin does not explicitly describe the detected object is a detected anomaly during a manufacturing process of a widget or that the rendering assists a decision-making process of a decision-making entity.
Sreejith teaches rendering superimposed images of a detected anomaly during a manufacturing process of a widget to assist a decision-making process of a decision-making entity (Sreejith; ¶0005, describes using an AI to produce an output including a classification of an anomaly (detected anomaly) in an image of a physical object and displaying the physical object on an AR display device, and “highlighting, on the display device concurrently with displaying the physical object, the anomaly.” Sreejith; ¶0003, describes an anomaly is an aspect of equipment that lies outside at least one engineering tolerance, or is a missing aspect of equipment that should be present and, Sreejith; ¶0002, describes manufacturers ensuring products (widgets) are withing engineering tolerances or is missing aspects of equipment that should be present (detected anomaly). Sreejith; ¶0027, further states this enables users to “more rapidly and more accurately identify anomalies in or on objects and, Sreejith; ¶0035, alerting a user when an anomaly has been highlighted (assist decision-making process of decision-making entity). In the combination, this teaches rendering superimposed images of a detected anomaly during a manufacturing process of a widget to the AR devices to assist a decision-making process of a decision-making entity.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the edge-assisted AR rendering system of Gruteser in view of Baldwin with the manufacturing detection and alerting of anomalies as taught by Sreejith to provide the benefit of faster and improved anomaly detection and/or inspection in an AR environment.
Claim 2, has similar limitations as of claim 9, therefore it is rejected under the same rationale as claim 9.
Claim 16, has similar limitations as of claim 9, therefore it is rejected under the same rationale as claim 9.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAN F KALHORI whose telephone number is (571)272-5475. The examiner can normally be reached Mon-Fri 8:30-5:30 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, DEVONA E FAULK can be reached at (571) 272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAN F KALHORI/Examiner, Art Unit 2618
/DEVONA E FAULK/Supervisory Patent Examiner, Art Unit 2618
Prosecution Insights