DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1-18 are currently pending and have been examined in this application. This communication is the first action on the merits (FAOM).
Examiner's Note
Examiner has cited particular paragraphs/columns and line numbers or figures in the
references as applied to the claims below for the convenience of the applicant. Although the
specified citations are representative of the teachings in the art and are applied to the specific
limitations within the individual claim, other passages and figures may apply as well. It is
respectfully requested from the applicant, in preparing the responses, to fully consider the
references in their entirety as potentially teaching all or part of the claimed invention, as well as
the context of the passage as taught by the prior art or disclosed by the examiner. Applicant is
reminded that the Examiner is entitled to give the broadest reasonable interpretation to the
language of the claims. Furthermore, the Examiner is not limited to Applicant's definition which is not specifically set forth in the disclosure.
Claim Objections
Claims 1, 3-4, 9-10, 12-13, and 17-18 are objected to because of the following informalities:
Claims 1 and 10 recite “any image collection device” but should instead recite --[[any]] an image collection device--.
Claims 3-4 and 12-13 recite “a processing order” but should instead recite --[[a]] the processing order--.
Claims 8 and 17 recite “pose information” but should instead recite --the pose information--.
Claims 9 and 18 recite “different time periods” in two instances, and the second instance should instead recite --the different time periods--.
Claims 9 and 18 recite “the familiar pixel” but should instead recite --the at least one familiar pixel--.
Appropriate correction is required.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they do not include the following reference sign(s) mentioned in the description: SLAM processor 100. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
Claim limitations “sampling unit”, “rendering unit”, “computation core”, “scheduler”, “single skip computation core”, “double skip computation core”, “2D sampling unit”, and “3D sampling unit” have been evaluated under the three-prong test set forth in MPEP § 2181, subsection I, but the result is inconclusive. Thus, it is unclear whether this limitation should be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitations use the word “means” or a generic placeholder coupled with functional language, but it is modified by some structure or material that is ambiguous regarding whether that structure or material is sufficient for performing the claimed function. The boundaries of this claim limitation are ambiguous; therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph. See 35 U.S.C. 112(b) rejections below for further details.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-18 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
As stated above, claims 1-9 include limitations “sampling unit”, “rendering unit”, “computation core”, “scheduler”, “single skip computation core”, “double skip computation core”, “2D sampling unit”, and “3D sampling unit” which have been evaluated under the three-prong test set forth in MPEP § 2181, subsection I, but the result is inconclusive. Thus, it is unclear whether this limitation should be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitations use the word “means” or a generic placeholder coupled with functional language, but it is modified by some structure or material that is ambiguous regarding whether that structure or material is sufficient for performing the claimed function. The boundaries of this claim limitation are ambiguous; therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
In response to this rejection, applicant must clarify whether this limitation should be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. Mere assertion regarding applicant’s intent to invoke or not invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph is insufficient. Applicant may:
(a) Amend the claim to clearly invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, by reciting “means” or a generic placeholder for means, or by reciting “step.” The “means,” generic placeholder, or “step” must be modified by functional language, and must not be modified by sufficient structure, material, or acts for performing the claimed function;
(b) Present a sufficient showing that 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, should apply because the claim limitation recites a function to be performed and does not recite sufficient structure, material, or acts to perform that function;
(c) Amend the claim to clearly avoid invoking 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, by deleting the function or by reciting sufficient structure, material or acts to perform the recited function; or
(d) Present a sufficient showing that 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, does not apply because the limitation does not recite a function or does recite a function along with sufficient structure, material or acts to perform that function.
Claim 1 recites “a sampling unit configured to hierarchically sample a two-dimensional (2D) image collected through any image collection device and pose information of the image collection device corresponding to the 2D image; and a rendering unit configured to perform SLAM through real-time rendering for data sampled by the sampling unit”. It is unclear if the “data sampled by the sampling unit” is the same or different from the “two-dimensional (2D) image” and the “pose information” which are previously recited as being sampled by the sampling unit. It is therefore unclear if the 2D image and pose information are used for the real-time rendering, or if different data is used for the real-time rendering. The metes and bounds of the claim language are vague and ill-defined, rendering the claim indefinite. As best understood, the claim will be interpreted broadly such that the data used for the real-time rendering includes at least the 2D image and the pose information.
The term “similar” in claims 6 and 16 is a relative term which renders the claim indefinite. The term “similar” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. As best understood, the claim will be interpreted broadly such that similar means that the workload is divided among the plurality of expert neural network operators, rather than any single expert neural network operator taking on the total workload.
Claims 7 and 16 recite “a sparse matrix” in three instances. It is unclear if these are all the same sparse matrix or different ones. The metes and bounds of the claim language are vague and ill-defined, rendering the claim indefinite. As best understood, the claim will be interpreted such that these are all the same sparse matrix.
Claim 10 recites “a sampling step of hierarchically sampling, by the SLAM processor, a 2D image collected through any image collection device and pose information of the image collection device corresponding to the 2D image; and a rendering step of performing, by the SLAM processor, SLAM through real-time rendering for data sampled in the sampling step”. It is unclear if the “data sampled in the sampling step” is the same or different from the “2D image” and the “pose information” which are previously recited as being sampled in the sampling step. It is therefore unclear if the 2D image and pose information are used for the real-time rendering, or if different data is used for the real-time rendering. The metes and bounds of the claim language are vague and ill-defined, rendering the claim indefinite. As best understood, the claim will be interpreted broadly such that the data used for the real-time rendering includes at least the 2D image and the pose information.
Claim 10 recites “a real-time neural network operation” and “a neural network operation”. It is unclear if these are referring to the same neural network operations, or if they are referring to different neural network operations. The metes and bounds of the claim language are vague and ill-defined, rendering the claim indefinite. As best understood, the claim will be interpreted broadly such that they are referring to the same neural network operations.
Claims 2-9 and 11-18 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being dependent on rejected claims 1 and 10 and for failing to cure the deficiencies listed above.
Allowable Subject Matter
Claims 1-18 would be allowable if rewritten or amended to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding independent claim 1 (and similar limitations of independent claim 10), it would not have been obvious to one of ordinary skill in the art to combine the teachings of the prior art in a meaningful way to arrive at the claim invention. The prior art fails to render obvious the following limitation in its entirety:
“a rendering unit configured to perform SLAM through real-time rendering for data sampled by the sampling unit, wherein the rendering unit comprises: a computation core configured to perform a neural network operation based on a sparse expert model including a plurality of expert neural networks and reducing a number of neural network channels by exclusively activating an expert neural network differently selected for each input batch; and a scheduler configured to schedule a processing order of input batches to improve computational efficiency of the computation core”
Lin (WO 2023/229600 A1) is the closest prior art of record, and teaches the following limitations: a processor (see at least [0007-0008, 0074] – one or more processors) for simultaneous localization and mapping (SLAM) (see at least abstract, [0036, 0050-0053] - SLAM) based on real-time neural network rendering (see at least [0029-0031, 0035, 0039] – data processing based on one or more neural networks… rendering virtual objects or creating mixed, virtual, or augmented reality content), the processor comprising: a sampling unit configured to hierarchically sample a two-dimensional (2D) image collected through any image collection device and pose information of the image collection device corresponding to the 2D image (see at least [0025, 0030-0032, 0035] – hierarchy loop…both video or static visual data captured by the camera and the inertial sensor data measured by the one or more inertial sensors are applied to determine and predict device poses… data processing module 228 and pose determination and prediction module 230); and a rendering unit configured to perform SLAM through real-time rendering for data sampled by the sampling unit (see at least [0029-0031, 0035, 0039] – data processing based on one or more neural networks… rendering virtual objects or creating mixed, virtual, or augmented reality content), wherein the rendering unit comprises: a computation core configured to perform a neural network operation (see at least [0043-0046] – a neural network (NN) is applied in a data processing model 250 to process content data (particularly, video and image data)).
Rouhani (US 2023/0316043 A1) teaches the following limitations: wherein the rendering unit comprises: a computation core configured to perform a neural network operation based on a sparse expert model including a plurality of expert neural networks and reducing a number of neural network channels by exclusively activating an expert neural network differently selected for each input batch (see at least [0042-0045, 0073] - Operating a machine learning model including one or more mixture of experts layers, such as machine learning system 100. When implemented, method 500 may allow for a mixture of balanced, fine grained sparsity and coarse grained sparsity that is inherent in MoE models... Designating one or more neural network experts in the mixture of experts layer to evaluate each input data shard); and a scheduler configured to schedule input batches (see at least [0076, 0098] - Scheduling each batch for processing by a neural network expert trained in a relevant modality. Scheduling each batch for processing by a neural network expert trained in a relevant modality may be performed by a reinforcement learning agent, and/or any suitable online or offline learning algorithm. In some examples, the reinforcement learning agent and/or other learning algorithm is trained in load-balancing. In this way, constraints are placed on the processing pathway. Some experts are eliminated from the process, and their potential inputs are preferentially direct towards other experts.).
Luk (US 2020/0226453 A1) teaches the following limitations: a scheduler configured to schedule a processing order of input batches to improve computational efficiency of the computation core (see at least [0030] - pre-compilation processor 104 to determine a dynamic batch schedule 122. The example dynamic batch schedule 122 is to configure the neural compute engine 108 to process different batches using the model 102. As used herein, a batch schedule and/or a batching schedule is a timeline utilized to schedule batches of input data for subsequent processing by the neural compute engine 108… the dynamic batch schedule 122 is indicative of an execution order of the batches, the execution order followed by the neural compute engine 108).
However, only by the use of impermissible hindsight would one of ordinary skill in the art be able to arrive at the claimed invention using the teachings of Lin, Rouhani, and Luk. The summarize, Lin teaches the concept of SLAM based on real-time neural network rendering by using images and pose information. However, Lin does not teach two main concepts recited in the claim. Lin does not teach (1) a sparse expert model including a plurality of expert neural networks and reducing a number of neural network channels by exclusively activating an expert neural network differently selected for each input batch and (2) a scheduler configured to schedule a processing order of input batches to improve computational efficiency of the computation core. Rouhani does teach (1) a sparse expert model including a plurality of expert neural networks and reducing a number of neural network channels by exclusively activating an expert neural network differently selected for each input batch. Rouhani also teaches a scheduler configured to schedule batches in a broad sense, but does not schedule a processing order of input batches to improve computational efficiency of the computation core as required by the claim. Luk teaches (2) a scheduler configured to schedule a processing order of input batches to improve computational efficiency of the computation core. Therefore, all of the limitations of the claim are disparately taught by Lin, Rouhani, and Luk. Although all the limitations are taught in the prior art, one of ordinary skill in the art must evaluate whether it would have been obvious to combine these teachings to arrive at the claimed invention.
Lin is directed towards SLAM rendering and utilizes a neural network to process input batches of images. Rouhani is directed to neural network processing for various domains or modalities corresponding to different machine learning tasks, such as speech recognition, image classification, machine translation, or parsing (see paragraph [0073] of Rouhani). In Rouhani, the input batches are sorted into batches of common modalities such as speech, images, language, text, etc., (see paragraph [0074] of Rouhani). The selected experts are trained in a relevant modality (see paragraphs [0075-0076] of Rouhani). Thus the core difference between Lin and Rouhani that makes this modification non-obvious is that all of Lin’s batches correspond to one modality (images) while Rouhani’s experts are selected differently for different modalities (speech, image, etc.,). There is lack of motivation to replace Lin’s neural network processing with Rouhani’s mixture of experts. There is no motivation to make this modification because there is a lack of evidence or suggestion that implementing the sparse mixture of experts into Lin’s neural network that processes a single kind of input batch would provide an improvement or solve a problem. Making this modification would also increase the complexity of Lin’s system because it would require new components to perform the selection of the mixture of experts and the scheduling, which are not simple substitutions and would entirely change how the image batches are processed by the neural network. These extra components and processes would increase processing time, which is considered to be undesirable in a real-time SLAM application such as Lin’s. Increasing processing time in a SLAM application negatively impacts the real-time rendering performance.
With regards to the scheduler, Rouhani teaches scheduling, but does not actually schedule an order of input batches to improve computational efficiency. Rouhani schedules each batch based on the expert trained in the relevant modality. In Luk, the input batches are scheduled in order to improve computational efficiency (see paragraph [0030] of Luk). In Luk, the input batch can correspond to an image (see paragraph [0076] of Luk). However, Luk does not even mention a mixture of experts, let alone a sparse mixture of experts. Luk also does not apply to the field of SLAM, rendering, or any similar application. It is unclear why one of ordinary skill in the art would schedule an order of input batches in a real-time SLAM rendering application because one of ordinary skill would expect in a real-time application that the images be processed in temporal order, not in a scheduled order that improves computational efficiency. Luk does not appear to be solving a problem by scheduling an order of batches that could be applied by one of ordinary skill in the art to a real-time SLAM rendering application.
Finally, it would not have been immediately obvious to one of ordinary skill in the art to combine the use of a sparse mixture of experts differently selected for each input batch with the use of a scheduler that schedules the order of input batches to improve computational efficiency.
Conclusion
The prior art made of record, and not relied upon, considered pertinent to applicant’s disclosure or directed to the state of art is listed on the enclosed PTO-892. The following is a brief description for relevant prior art that was cited but not applied: Shi (US 2024/0029300 A1) and Ahmed (US 2025/0014200 A1).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CAITLIN MCCLEARY whose telephone number is (703)756-1674. The examiner can normally be reached Monday - Friday 10:00 am - 7:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Navid Z Mehdizadeh can be reached at (571) 272-7691. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CAITLIN R MCCLEARY/Examiner, Art Unit 3669
Prosecution Insights