DETAILED ACTION
I. Introduction
This Office action addresses U.S. reissue application number 19/060,972 (“’972 reissue application” or “instant application”), having a filing date of 24 February 2025. Because the instant application was filed on or after September 16, 2012, the statutory provisions of the America Invents Act (“AIA ”) will govern this proceeding.
The instant application is a reissue of U.S. Patent 11,590,658 (“’658 patent”) titled “TACTILE INFORMATION ESTIMATION APPARATUS, TACTILE INFORMATION ESTIMATION METHOD, AND PROGRAM”, which issued to Kuniyuki Takahashi et al. on 28 February 2023 with claims 1-34 (“issued claims”). The application resulting in the ‘658 patent was filed on 6 December 2019 and assigned U.S. patent application number 16/706,478 (“’478 application”).
II. Other Proceedings
After review of Applicant’s statements as set forth in the instant application, and the examiner's independent review of the ‘658 patent itself and its prosecution history, the examiner has failed to locate any current ongoing litigation. The examiner has likewise failed to locate any previous or current reexaminations (ex parte or inter partes), supplemental examinations, or other post issuance proceedings.
III. Priority
The ‘478 application is a continuation of application PCT/JP2019/000971 (“the PCT application”), filed 15 January 2019.
The ‘478 application also claims foreign priority under 35 U.S.C. § 119(a)-(d) to Japanese Application JP2018-005151, filed 16 January 2018.
As a reissue application, the instant application is entitled to the priority date of the ‘658 patent, the patent being reissued. Thus, the instant reissue application has a priority date of at least 15 January 2019, the filing date of the PCT application. The priority date could be as early as 16 January 2018, depending upon the specific subject matter of the claim.
The Office acknowledges the priority claim under 35 U.S.C. § 119(a)-(d), but notes that the priority claim has not been perfected through the filing of certified English language translations of the Japanese priority document, which would establish enablement and written description support for the claims, as well as establishing the precise priority date for any given feature/limitation/claim. See MPEP § 2136(a)(II).
The priority date will be determined on a claim-by-claim basis, as necessary.
Because the effective filing date of the instant application is after to March 16, 2013, the pre-AIA ‘First to Invent’ provisions do not apply. Instead, the AIA First Inventor to File (“AIA -FITF”) provisions will apply.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
IV. Claim Construction
During examination, claims are given the broadest reasonable interpretation consistent with the specification and limitations in the specification are not read into the claims. See MPEP § 2111 et seq.
Upon review of the original specification and prosecution history, the examiner has found no instances of lexicographic definitions, either express or implied, that are inconsistent with the ordinary and customary meaning of the respective terms. Therefore, for the purposes of claim construction, the examiner concludes that there are no claim terms for which applicant is acting as their own lexicographer. See MPEP § 2111.01(IV).
If applicant intended lexicographic definitions that have not been identified as such by the examiner, they are asked to note the term and the location in the specification or prosecution history supporting the lexicographic definition in response to this Office action.
Additionally, upon review of the pending claims, the examiner finds no instances where the claim terms explicitly include functional language which invokes the provisions of 35 U.S.C. § 112(f) or pre-AIA 35 U.S.C. § 112, sixth paragraph.
V. Information Disclosure Statement
Applicant’s Information Disclosure Statement (IDS), filed 24 February 2025, has been received and entered into the record. Since the Information Disclosure Statement complies with the provisions of MPEP § 609, the references cited therein have been considered by the examiner.
It is noted that foreign language document A18 was not considered, because the only copy of this document located in the file wrapper of the ‘478 application was in the Japanese language.
It is further noted that the copy of non-patent literature document A27 submitted during prosecution of the ‘478 application was illegible, so a new copy is entered into the record with the instant Office action.
See attached form PTO-1449.
VI. Preliminary Amendment
Applicant’s preliminary amendment, filed 24 February 2025, has been received and entered into the record. The preliminary amendment included amendments to the specification and claims.
Specifically, claims 16, 19, and 34 have been amended. Claims 1-34 remain pending in the application.
VII. Reissue Declaration
The reissue oath/declaration filed with this application is defective (see 37 CFR 1.175 and MPEP § 1414) because of the following:
The reissue declaration filed with the instant reissue application indicates that the application for the original patent was filed under 37 C.F.R. § 1.46 by the assignee of the entire interest. However, the reissue declaration submitted was on a form PTO/AIA /05, Reissue Application Declaration by the Inventor. When the reissue applicant is a juristic entity, as is the case here, then the reissue declaration should be filed on form PTO/AIA /06, Reissue Application Declaration by the Assignee. Furthermore, the reissue declaration must be signed by an official of the applicant who has a title that carries apparent authority, or someone who makes a statement of authorization to act (e.g., an employee of the assignee who by corporate resolution of a Board of Directors has been given authority to act on behalf of the juristic entity). See MPEP § 325.
In addition, the reissue declaration submitted included the name of only one of the inventors. A reissue declaration must include the names of all inventors. See MPEP § 1414.01 and 37 C.F.R. § 1.46(b)(4).
VIII. Rejections under 35 U.S.C. § 251
Claims 1-34 are rejected as being based upon a defective reissue declaration under 35 U.S.C. 251 as set forth above. See 37 CFR 1.175.
The nature of the defect(s) in the declaration is set forth in the discussion above in this Office action.
IX. Recapture under 35 U.S.C. § 251
In In re Clement, 131 F.3d at 1468-70, 45 USPQ2d at 1164-65, the Court of Appeals for the Federal Circuit set forth a three-step test for recapture analysis. In North American Container, 415 F.3d at 1349, 75 USPQ2d at 1556, the court restated this test as follows:
We apply the recapture rule as a three-step process:
(1) first, we determine whether, and in what respect, the reissue claims are broader in scope than the original patent claims;
(2) next, we determine whether the broader aspects of the reissue claims relate to subject matter surrendered in the original prosecution; and
(3) finally, we determine whether the reissue claims were materially narrowed in other respects, so that the claims may not have been enlarged, and hence avoid the recapture rule.
Step 1
With respect to step 1 (see MPEP § 1412.02(I)(A)), applicants seek to broaden reissue claims 16-34 by at least deleting/omitting the following limitations which are present in issued independent claims 1, 14, 16, and 34 of the ‘658 patent, but not in reissue claims 16 and 34 (paraphrased):
extract[ing] and output[ting] tactile information.
Step 2
With respect to step 2 (see MPEP § 1412.02(I)(B)), there was an instance where applicants surrendered subject matter during prosecution of the original application (which became the patent to be reissued). Note that the "original application" includes the patent family’s entire prosecution history. MBO Laboratories, Inc. v. Becton, Dickinson & Co., 602 F.3d 1306, 94 USPQ2d 1598 (Fed. Cir. 2010).
During prosecution of the ‘478 application, applicant filed a response to a first non-final rejection on 21 June 2022. Therein, applicant submitted a new claim set, and argued that the following limitations distinguished over the prior art of record (see page 17):
a configuration that inputs visual information
a configuration that outputs tactile information
The examiner subsequently issued a notice of allowance.
As noted in MPEP § 1412.02, “If an original patent claim limitation now being omitted or broadened in the present reissue application was originally relied upon by applicant in the original application to make the claims allowable over the art, the omitted limitation relates to subject matter previously surrendered by applicant. The reliance by applicant to define the original patent claims over the art can be by presentation of new/amended claims to define over the art, or an argument/statement by applicant that a limitation of the claim(s) defines over the art.”
With respect to whether applicant surrendered any subject matter, it is to be noted that a patent owner (reissue applicant) is bound by the argument that applicant relied upon to overcome, for example, an art rejection in the original application for the patent to be reissued, regardless of whether the Office adopted the argument in allowing the claims. Greenliant Systems, Inc. v. Xicor LLC, 692 F.3d 1261, 1271, 103 USPQ2d 1951, 1958 (Fed. Cir. 2012). As pointed out by the court, "[i]t does not matter whether the examiner or the Board adopted a certain argument for allowance; the sole question is whether the argument was made." Id.
Given the above-cited surrendered subject matter, and in consideration of the subject matter omitted in independent reissue claims 16 and 34 (cited above in step 1 of the analysis), the Office concludes that the following broader aspects of reissue claims 16-34 are related to the subject matter surrendered during original prosecution of the ‘478 application:
output[ting] tactile information.
The limitation of outputting tactile information has been entirely eliminated from independent reissue claims 16 and 34.
Step 3
With respect to step 3 (see MPEP § 1412.02(I)(C)), the Office has reviewed and analyzed reissue independent claims 16 and 34, and concluded that there has been no material narrowing of the reissue claims in such a way that recapture has been avoided. Therefore, in view of the surrendered subject matter that has been broadened in the reissue claims, claims 16 and 34 are subject to rejection under 35 U.S.C. § 251.
Dependent claims 17-33 are likewise rejected, as they fail to restore all of the surrendered subject matter to the claims.
Claims 16-34 are therefore rejected under 35 U.S.C. § 251 as being an impermissible recapture of broadened claimed subject matter surrendered in the application for the patent upon which the present reissue is based. See Greenliant Systems, Inc. et al v. Xicor LLC, 692 F.3d 1261, 103 USPQ2d 1951 (Fed. Cir. 2012); In re Shahram Mostafazadeh and Joseph O. Smith, 643 F.3d 1353, 98 USPQ2d 1639 (Fed. Cir. 2011); North American Container, Inc. v. Plastipak Packaging, Inc., 415 F.3d 1335, 75 USPQ2d 1545 (Fed. Cir. 2005); Pannu v. Storz Instruments Inc., 258 F.3d 1366, 59 USPQ2d 1597 (Fed. Cir. 2001); Hester Industries, Inc. v. Stein, Inc., 142 F.3d 1472, 46 USPQ2d 1641 (Fed. Cir. 1998); In re Clement, 131 F.3d 1464, 45 USPQ2d 1161 (Fed. Cir. 1997); Ball Corp. v. United States, 729 F.2d 1429, 1436, 221 USPQ 289, 295 (Fed. Cir. 1984).
The reissue application contains claim(s) that are broader than the issued patent claims. The record of the application for the patent shows that the broadening aspect (in the reissue) relates to claimed subject matter that applicant previously surrendered during the prosecution of the application. Accordingly, the narrow scope of the claims in the patent was not an error within the meaning of 35 U.S.C. § 251, and the broader scope of claim subject matter surrendered in the application for the patent cannot be recaptured by the filing of the present reissue application.
X. Specification
The specification is objected to as failing to provide proper antecedent basis for the claimed subject matter. See 37 CFR 1.75(d)(1) and MPEP § 608.01(o). Correction of the following is required:
Claims 16-34 include the term “end effectors.” However, this term does not appear in any form in applicant’s disclosure.
XI. Claim Rejections - 35 U.S.C. § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-15 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 1
Under Step 1, Claim 1 is a machine/apparatus claim.
The claim includes one or more memories and one or more processors configured to perform two steps.
The first step is to input visual information of a first object into a neural network model. The second step is to extract and output, based on the neural network model, information relating to tactile information of the first object.
Under Step 2A, Prong One, these limitations are directed to the abstract idea of a mental process.
The extraction process utilizes a neural network model to analyze the input visual information and to determine analogous tactile information consistent with the input visual information.
Under the broadest reasonable interpretation, the “extracting” encompasses mental observations or evaluations that are practically performed in the human mind. For example, the claimed extracting information relating to tactile information of the first object encompasses observing data and performing an evaluation by comparing data within the neural network model (i.e., observation, evaluation, judgement, and opinion) to find analogous tactile information that corresponds to the input visual information. See MPEP §2106.04(a)(2), subsection III.
Under Step 2A, Prong Two, the “input” and “output” limitations are mere data gathering and output recited at a high level of generality, and are thus insignificant extra-solution activity. See MPEP § 2106.05(g).
The limitation “the neural network model having been trained using visual information of a second object and tactile information of the second object” merely modifies the neural network model, and does not integrate the judicial exception into a practical application of the exception.
Similarly, the limitation “wherein the visual information of the first object includes at least texture information of the surface of the first object and the visual information of the second object includes at least texture information of the surface of the second object” merely modifies the type of information used to train the neural network model and input to the neural network model for analysis. It likewise fails to integrate the judicial exception into a practical application of the exception.
The “extract” limitation is recited as being performed by a processor. However, the processor is likewise recited at a high level of generality, performing the generic functions of input and output of data, and performing the abstract idea of extracting information as discussed above with respect to Prong One, such that it amounts to no more than mere instructions to apply the exception using a generic processor.
The limitation of “based on the neural network model” provides nothing more than mere instructions to implement an abstract idea on a generic processor. See MPEP § 2106.05(f). MPEP § 2106.05(f) provides the following considerations for determining whether a claim simply recites a judicial exception with the words “apply it” (or an equivalent), such as mere instructions to implement an abstract idea on a computer: (1) whether the claim recites only the idea of a solution or outcome i.e., the claim fails to recite details of how a solution to a problem is accomplished; (2) whether the claim invokes computers or other machinery merely as a tool to perform an existing process; and (3) the particularity or generality of the application of the judicial exception.
The judicial exception of “extracting information relating to tactile information of the first object” is performed “based on the neural network model.” The neural network model is used to generally apply the abstract idea without placing any limits on how the neural network model functions. Rather, these limitations only recite the outcome of “extracting information relating to tactile information of the first object” and do not include any details about how the “extracting” is accomplished. See MPEP § 2106.05(f).
The recitation of “based on the neural network model” also merely indicates a field of use or technological environment in which the judicial exception is performed. Although the additional element “based on the neural network model” limits the identified judicial exception “extracting information relating to tactile information of the first object,” this type of limitation merely confines the use of the abstract idea to a particular technological environment (neural networks) and thus fails to add an inventive concept to the claims. See MPEP § 2106.05(h). Even when viewed in combination, these additional elements do not integrate the recited judicial exception into a practical application, and the claim is directed to the judicial exception.
Under Step 2B, this part of the eligibility analysis evaluates whether the claim as a whole amounts to significantly more than the recited exception i.e., whether any additional element, or combination of additional elements, adds an inventive concept to the claim. See MPEP § 2106.05.
As explained with respect to Step 2A, Prong Two, the additional element of “based on the neural network model” is at best mere instructions to “apply” the abstract ideas, which cannot provide an inventive concept. See MPEP § 2106.05(f).
Additional elements “input” and “output” were both found to be insignificant extra-solution activity in Step 2A, Prong Two, because they were determined to be insignificant limitations as necessary data gathering and outputting. However, a conclusion that an additional element is insignificant extra solution activity in Step 2A, Prong Two should be re-evaluated in Step 2B. See MPEP § 2106.05, subsection I.A. At Step 2B, the evaluation of the insignificant extra-solution activity consideration takes into account whether or not the extra-solution activity is well understood, routine, and conventional in the field. See MPEP § 2106.05(g).
As discussed in Step 2A, Prong Two above, the recitations of “input at least visual information of a first object” and “output information relating to tactile information of the first object” are recited at a high level of generality. These elements amount to receiving or transmitting data over a network and are well understood, routine, conventional activity. See MPEP §2106.05(d), subsection II.
As discussed in Step 2A, Prong Two above, the recitation of a processor to perform the “input,” “extract,” and “output” steps amounts to no more than mere instructions to apply the exception using a generic computer component. Even when considered in combination, these additional elements represent mere instructions to implement an abstract idea or other exception on a computer and insignificant extra-solution activity, which do not provide an inventive concept.
Dependent claims 2-12 do not incorporate additional features that would change the above analysis, and are likewise rejected.
Claim 14 is a method/process claim having analogous limitations to that of claim 1, and is likewise rejected for being directed to an abstract idea without significantly more.
Claim 13
Under Step 1, Claim 13 is a machine/apparatus claim.
The claim includes one or more memories and one or more processors configured to perform one step.
The claimed step is to learn a neural network model.
Under Step 2A, Prong One, this limitation is directed to the abstract idea of a mental process.
The learning process utilizes visual information and tactile information to form a neural network model.
Under the broadest reasonable interpretation, the “learning” encompasses mathematical concepts. For example, applicant’s specification includes discussion of the application of numerous mathematical functions and calculations in generating the model based upon received visual and tactile information (see, e.g., col. 5, line 14 through col. 9, line 63. See MPEP §2106.04(a)(2), subsection I.
Under Step 2A, Prong Two, the “input” and “output” limitations are mere data gathering and output recited at a high level of generality, and are thus insignificant extra-solution activity. See MPEP § 2106.05(g).
The limitation “using visual information of a second object and tactile information of the second object” merely modifies the data that is used to train the neural network model, and does not integrate the judicial exception into a practical application of the exception.
Similarly, the limitation “wherein the visual information of the first object includes at least texture information of the surface of the first object and the visual information of the second object includes at least texture information of the surface of the second object” merely modifies the type of information used to train the neural network model and input to the neural network model for analysis. It likewise fails to integrate the judicial exception into a practical application of the exception.
The “learn” limitation is recited as being performed by a processor. However, the processor is likewise recited at a high level of generality, performing the generic functions of input and output of data, and performing the abstract idea of learning a neural network model as discussed above with respect to Prong One, such that it amounts to no more than mere instructions to apply the exception using a generic processor.
The additional elements do not integrate the recited judicial exception into a practical application, and the claim is directed to the judicial exception.
Under Step 2B, this part of the eligibility analysis evaluates whether the claim as a whole amounts to significantly more than the recited exception i.e., whether any additional element, or combination of additional elements, adds an inventive concept to the claim. See MPEP § 2106.05.
As explained with respect to Step 2A, Prong Two, additional elements “input” and “output” were both found to be insignificant extra-solution activity in Step 2A, Prong Two, because they were determined to be insignificant limitations as necessary data gathering and outputting. However, a conclusion that an additional element is insignificant extra solution activity in Step 2A, Prong Two should be re-evaluated in Step 2B. See MPEP § 2106.05, subsection I.A. At Step 2B, the evaluation of the insignificant extra-solution activity consideration takes into account whether or not the extra-solution activity is well understood, routine, and conventional in the field. See MPEP § 2106.05(g).
As discussed in Step 2A, Prong Two above, the recitations of “at least visual information of a first object is inputted” and “the learned neural network model outputs information relating to tactile information of the first object” are recited at a high level of generality. These elements amount to receiving or transmitting data over a network and are well understood, routine, conventional activity. See MPEP §2106.05(d), subsection II.
As discussed in Step 2A, Prong Two above, the recitation of a processor to perform the “input,” “learn,” and “output” steps amounts to no more than mere instructions to apply the exception using a generic computer component. Even when considered in combination, these additional elements represent mere instructions to implement an abstract idea or other exception on a computer and insignificant extra-solution activity, which do not provide an inventive concept.
Claim 15 is a method/process claim having analogous limitations to that of claim 13, and is likewise rejected for being directed to an abstract idea without significantly more.
Contrast claims 1-15 with independent claims 16 and 34, both of which include the limitation of utilizing the extracted information to control the one or more end effectors, thus integrating the judicial exception into a practical application.
Applicant’s attention is directed to the Federal Register Notice of 17 July 2024 “2024 Guidance Update on Patent Subject Matter Eligibility, Including on Artificial Intelligence.”
XII. Claim Rejections - 35 U.S.C. § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-34 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Specifically, claims 1-34 include the limitation of a neural network model. However, while applicant’s specification does contain support for a generalized “model,” and discloses the use of a convolutional neural network (CNN) and a feedforward neural network (FNN) (see col. 3, lines 39-49), there is no support for the claimed generalized “neural network model.”
In addition, claims 16-34 include the use of “end effectors.” However, applicant’s specification includes support only for the use of “grippers.” As evidenced by two non-patent documents12, the scope of the term “end effector” is considerably broader than the term “gripper,” and applicant’s disclosure fails to provide support for the broader “end effector” that is claimed.
The Office acknowledges that the originally-filed ‘478 application included claims 22-34 that were directed to the use of end effectors.
It is well accepted that a satisfactory description may be found in originally-filed claims or any other portion of the originally-filed specification. See In re Koller, 613 F.2d 819, 204 USPQ 702 (CCPA 1980); In re Gardner, 475 F.2d 1389, 177 USPQ 396 (CCPA 1973); In re Wertheim, 541 F.2d 257, 191 USPQ 90 (CCPA 1976). However, that does not mean that all originally-filed claims have adequate written support. The specification must still be examined to assess whether an originally-filed claim has adequate support in the written disclosure and/or the drawings. Issues of adequate written description may arise for original claims, for example, when an aspect of the claimed invention has not been described with sufficient particularity such that one skilled in the art would recognize that the inventor had possession of the claimed invention at the time of filing3.
Upon review of applicant’s disclosure, the Office has concluded that there is sufficient support for an embodiment wherein grippers are utilized. What is not supported is an embodiment wherein end effectors are utilized.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3, 5, 7, 8, and 16-33 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 3, this claim includes the limitation “the neural network.” There is no antecedent basis for this limitation.
Regarding claim 5, this claim includes the limitation of “output a result of mapping a visual and tactile feature amount of the first object to a space of two dimensions or higher.” This claim is indefinite, because the claim outputs the result of a mapping function that is not itself claimed.
Regarding claim 7, this claim includes the limitation “the information.” There are a number of different types of “information” in parent claim 1, rendering the precise interpretation of the term unclear. It is recommended that applicant amend the claim to refer instead to “the extracted information,” as in analogous claim 19.
Regarding claims 8 and 25, these claims include the limitation “the tactile information of the second object acquired by one or more tactile sensors.” There is no antecedent basis for this claim limitation.
Regarding claim 16, this claim is directed to a system. However, the claim includes the limitations “one or more visual sensors acquiring visual information of a first object” and “one or more effectors manipulating the first object” (emphasis added).
The presence of method steps within an apparatus claim renders the claim indefinite, since it is unclear whether the claim would be infringed by an apparatus capable of carrying out the claimed method steps, or if the actual execution of the method steps would be necessary for infringement.
Claims 17-33, depending from claim 16, are likewise rejected.
Similarly, claim 27 includes the limitation “one or more tactile sensors acquiring tactile information of the first object.” This claim is rejected for the reasons discussed above with respect to claim 16.
Regarding claim 18, this claim includes the limitation “the neural network.” There is no antecedent basis for this claim limitation.
Regarding claim 30, this claim includes the limitation “reinforce learning.” The meaning of this term is unclear, rendering the claim indefinite.
Applicant may have intended to refer to the “reinforcement learning” that is disclosed in applicant’s specification.
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claims 11 and 24 are rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends.
Specifically, claims 11 and 24 include the limitation that the visual information of the first object includes an image of the first object, and visual information of the second object includes an image of the second object. Since by definition, visual information includes image information, this claim fails to further limit its parent claim.
Applicant may cancel the claim(s), amend the claim(s) to place the claim(s) in proper dependent form, rewrite the claim(s) in independent form, or present a sufficient showing that the dependent claim(s) complies with the statutory requirements.
XIII. Original Patent Requirement
Claims 1-34 are rejected under 35 U.S.C. § 251 for failing the original patent requirement.
To satisfy the original patent requirement where a new invention is sought by reissue, "… the specification must clearly and unequivocally disclose the newly claimed invention as a separate invention." Antares Pharma, Inc., 771 F.3d at 1363, 112 USPQ2d at 1871.
Claims 1-34 include the limitation “neural network model.” While applicant’s disclosure supports a generalized “model”, as well as a “Convolutional Neural Network (CNN)” or a “Feedforward Neural Network (FNN)”, there is no support for the claimed generalized “neural network model,” as discussed above with respect to 35 U.S.C. § 112(a). Claims 1-34 are rejected as failing the original patent requirement on the same basis.
In addition, the embodiment of claims 16-34, which includes the use of end effectors, is not supported by applicant’s specification, as discussed above with respect to 35 U.S.C. § 112(a). Claims 16-34 are rejected as failing the original patent requirement on the same basis.
XIV - Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 4-8, 11, 13-17, 19, 21-25, 29-32, and 34 are rejected under 35 U.S.C. § 102(a)(1) as being anticipated by “A Bio-inspired Neural Sensory-Motor Coordination Scheme for Robot Reaching and Preshaping” by Cecilia Laschi et al. (“Laschi”).
Claim 1
With respect to claim 1, Laschi discloses an apparatus for estimating tactile information as claimed, comprising:
a) one or more memories (inherent in a computerized robotic control system);
b) one or more processors (inherent in a computerized robotic control system) configured to:
i) input at least visual information of a first object into a neural network model (see disclosure that the Tactile Prediction Module receives as input the geometric features of the object provided by the Vision Module, Section III.D, page 4, col. 1), the neural network model having been trained using visual information of a second object and tactile information of the second object (see disclosure that during training of the neural network, the system learns correlations between visual information, hand and arm configurations, and tactile images, Section II, page 2, col. 1, last full paragraph); and
ii) extract and output, based on the neural network model, information relating to tactile information of the first object (see disclosure that the Tactile Prediction Module receives information about the object geometric features from the Vision Module and information from the Preshaping Module to the hand/arm configuration, and based on this information, the Tactile Prediction Module provides as output the tactile image expected when the object is contacted, Section II, page 2, col. 1, fourth paragraph);
c) wherein the visual information of the first object includes at least texture information of a surface of the first object, and the visual information of the second object includes at least texture information of the second object (see disclosure that during training the system learns correlations between visual information, hand and arm configurations, and tactile images, Section II, page 2, col. 1, last full paragraph; see also disclosure that the Tactile Prediction Module provides as output the tactile image expected when the object is contacted, Section II, page 2, col. 1, fourth paragraph; see also disclosure that the tactile image expected includes an estimation of object surface properties [i.e., texture information], Section II, first paragraph, page 1, col. 2).
Claim 4
With respect to claim 4, Laschi discloses the apparatus of claim 1, wherein the information is extracted based on an output from an intermediate layer of the neural network model (see disclosure that the output of the SANFIS neural network is based on outputs of intermediate layers, Section III.A, pages 2-3, and illustrated in Fig. 3:
PNG
media_image1.png
350
526
media_image1.png
Greyscale
Laschi, Figure 3
Claim 5
With respect to claim 5, Laschi discloses the apparatus of claim 1, wherein the one or more processors are further configured to output a result of mapping a visual and tactile feature amount of the first object to a space of two dimensions of higher (see disclosure that object position (POS1, POS2) is represented as the 2D position of the centroid of a bounding box enclosing the shape of the object in the camera reference system, Section III.B, page 3, col. 2, first paragraph).
Claim 6
With respect to claim 6, Laschi discloses the apparatus of claim 1, wherein the information is extracted based on a visual and tactile feature amount generated by the neural network model (see disclosure that the Tactile Prediction Module receives information about the object geometric features from the Vision Module and information from the Preshaping Module to the hand/arm configuration, and based on this information, the Tactile Prediction Module provides as output the tactile image [i.e., a tactile feature amount] expected when the object is contacted, Section II, page 2, col. 1, fourth paragraph).
Claim 7
With respect to claim 7, Laschi discloses the apparatus of claim 1, wherein the information includes at least one of the tactile information of the first object or property information of the first object (see disclosure that the Tactile Prediction Module receives information about the object geometric features from the Vision Module and information from the Preshaping Module to the hand/arm configuration, and based on this information, the Tactile Prediction Module provides as output the tactile image expected when the object is contacted, Section II, page 2, col. 1, fourth paragraph).
Claim 8
With respect to claim 8, Laschi discloses the apparatus of claim 1, wherein the neural network model is trained using the tactile information of the second object acquired by one or more tactile sensors (see disclosure of the use of a humanoid robot that includes a hand with tactile sensors for grasping tasks, Section II, page 2, col. 1, first paragraph; see also disclosure that during a training phase, the system grasps different kinds of objects in different positions to learn correlations between visual information, hand and arm configurations, and tactile images, Section II, page 2, col. 1, last full paragraph).
Claim 11
With respect to claim 11, Laschi discloses the apparatus of claim 1, wherein the visual information of the first object includes an image of the first object, and the visual information of the second object includes an image of the second object (see disclosure that the Vision Module receives binocular images of the scene acquired by cameras in the robot head and provides information about geometric features of the object of interest, such as object shape, dimensions, position, and orientation, Section II, page 2, col. 1, second paragraph).
Claim 13
With respect to claim 13, Laschi discloses an apparatus for learning a neural network model as claimed, comprising:
a) one or more memories (inherent in a computerized robotic control system);
b) one or more processors (inherent in a computerized robotic control system) configured to:
i) learn the neural network model using visual information of a second object and tactile information of the second object (see disclosure that during training of the neural network, the system learns correlations between visual information, hand and arm configurations, and tactile images, Section II, page 2, col. 1, last full paragraph);
c) wherein when at least visual information of a first object is inputted to the learned neural network model, the learned neural network model outputs information relating to tactile information of the first object (see disclosure that the Tactile Prediction Module receives information about the object geometric features from the Vision Module and information from the Preshaping Module to the hand/arm configuration, and based on this information, the Tactile Prediction Module provides as output the tactile image expected when the object is contacted, Section II, page 2, col. 1, fourth paragraph); and
d) wherein the visual information of the first object includes at least texture information of a surface of the first object, and the visual information of the second object includes at least texture information of the second object (see disclosure that during training the system learns correlations between visual information, hand and arm configurations, and tactile images, Section II, page 2, col. 1, last full paragraph; see also disclosure that the Tactile Prediction Module provides as output the tactile image expected when the object is contacted, Section II, page 2, col. 1, fourth paragraph; see also disclosure that the tactile image expected includes an estimation of object surface properties [i.e., texture information], Section II, first paragraph, page 1, col. 2).
Claim 14
With respect to claim 14, Laschi discloses a method for estimating tactile information as claimed, comprising:
a) inputting, by one or more processors (inherent in a computerized robotic control system), at least visual information of a first object into a neural network model (see disclosure that the Tactile Prediction Module receives as input the geometric features of the object provided by the Vision Module, Section III.D, page 4, col. 1), the neural network model having been trained using visual information of a second object and tactile information of the second object (see disclosure that during training of the neural network, the system learns correlations between visual information, hand and arm configurations, and tactile images, Section II, page 2, col. 1, last full paragraph); and
b) extracting and outputting, by the one or more processors (inherent in a computerized robotic control system), based on the neural network model, information relating to tactile information of the first object (see disclosure that the Tactile Prediction Module receives information about the object geometric features from the Vision Module and information from the Preshaping Module to the hand/arm configuration, and based on this information, the Tactile Prediction Module provides as output the tactile image expected when the object is contacted, Section II, page 2, col. 1, fourth paragraph);
c) wherein the visual information of the first object includes at least texture information of a surface of the first object, and the visual information of the second object includes at least texture information of the second object (see disclosure that during training the system learns correlations between visual information, hand and arm configurations, and tactile images, Section II, page 2, col. 1, last full paragraph; see also disclosure that the Tactile Prediction Module provides as output the tactile image expected when the object is contacted, Section II, page 2, col. 1, fourth paragraph; see also disclosure that the tactile image expected includes an estimation of object surface properties [i.e., texture information], Section II, first paragraph, page 1, col. 2).
Claim 15
With respect to claim 15, Laschi discloses a method for learning a neural network model as claimed, comprising:
a) learning, by one or more processors (inherent in a computerized robotic control system), the neural network model using visual information of a second object and tactile information of the second object (see disclosure that during training of the neural network, the system learns correlations between visual information, hand and arm configurations, and tactile images, Section II, page 2, col. 1, last full paragraph);
b) wherein when at least visual information of a first object is inputted to the learned neural network model, the learned neural network model outputs information relating to tactile information of the first object (see disclosure that the Tactile Prediction Module receives information about the object geometric features from the Vision Module and information from the Preshaping Module to the hand/arm configuration, and based on this information, the Tactile Prediction Module provides as output the tactile image expected when the object is contacted, Section II, page 2, col. 1, fourth paragraph); and
c) wherein the visual information of the first object includes at least texture information of a surface of the first object, and the visual information of the second object includes at least texture information of the second object (see disclosure that during training the system learns correlations between visual information, hand and arm configurations, and tactile images, Section II, page 2, col. 1, last full paragraph; see also disclosure that the Tactile Prediction Module provides as output the tactile image expected when the object is contacted, Section II, page 2, col. 1, fourth paragraph; see also disclosure that the tactile image expected includes an estimation of object surface properties [i.e., texture information], Section II, first paragraph, page 1, col. 2).
Claim 16
With respect to claim 16, Laschi discloses a system as claimed, comprising:
a) one or more memories (inherent in a computerized robotic control system);
b) one or more visual sensors acquiring visual information of a first object (see disclosure that the Vision Module receives binocular images of the scene acquired by cameras in the robot head and provides information about geometric features of the object of interest, such as object shape, dimensions, position, and orientation, Section II, page 2, col. 1, second paragraph);
c) one or more end effectors manipulating the first object (see Figs. 10 and 12, as well as disclosure that after a learning phase, the system starts from visual data, calculates the position and orientation of the hand for grasping, selects the best-suited hand configuration, and predicts the tactile feedback after grasping, Abstract); and
d) one or more processors (inherent in a computerized robotic control system) configured to:
i) input at least visual information of a first object into a neural network model (see disclosure that the Tactile Prediction Module receives as input the geometric features of the object provided by the Vision Module, Section III.D, page 4, col. 1), and extract, based on the neural network model, information to control the one or more end effectors (see disclosure that the Tactile Prediction Module receives information about the object geometric features from the Vision Module and information from the Preshaping Module to the hand/arm configuration, and based on this information, the Tactile Prediction Module provides as output the tactile image expected when the object is contacted, Section II, page 2, col. 1, fourth paragraph); and
ii) control the one or more end effectors to manipulate the first object based on the extracted information (see disclosure that after a learning phase, the system starts from visual data, calculates the position and orientation of the hand for grasping, selects the best-suited hand configuration, and predicts the tactile feedback after grasping, Abstract; see also disclosure of the evaluation of the neural network in controlling the grasping of the robot, Sections IV.A and IV.B, page 5, col. 1, last paragraph through page 6, col. 1);
e) wherein the neural network model has been trained using visual information of a second object and tactile information of the second object (see disclosure that during training of the neural network, the system learns correlations between visual information, hand and arm configurations, and tactile images, Section II, page 2, col. 1, last full paragraph); and
f) wherein the visual information of the first object includes at least texture information of a surface of the first object, and the visual information of the second object includes at least texture information of the second object (see disclosure that during training the system learns correlations between visual information, hand and arm configurations, and tactile images, Section II, page 2, col. 1, last full paragraph; see also disclosure that the Tactile Prediction Module provides as output the tactile image expected when the object is contacted, Section II, page 2, col. 1, fourth paragraph; see also disclosure that the tactile image expected includes an estimation of object surface properties [i.e., texture information], Section II, first paragraph, page 1, col. 2).
Claim 19
With respect to claim 19, Laschi discloses the system of claim 16, wherein the extracted information includes at least one of tactile information of the first object, property information of the first object, or a visual and tactile feature amount of the first object (see disclosure that the Tactile Prediction Module receives information about the object geometric features from the Vision Module and information from the Preshaping Module to the hand/arm configuration, and based on this information, the Tactile Prediction Module provides as output the tactile image expected when the object is contacted, Section II, page 2, col. 1, fourth paragraph).
Claim 21
With respect to claim 21, Laschi discloses the system of claim 16, wherein the one or more processors are configured to extract the information by inputting at least the visual information of the first object into the neural network model while the one or more end effectors do not manipulate the first object (see disclosure that the Tactile Prediction Module receives as input the geometric features of the object provided by the Vision Module, Section III.D, page 4, col. 1; see also disclosure that the robot hand grasping action takes place based upon a calculation of the position and orientation of the hand for grasping, Abstract).
Claim 22
With respect to claim 22, Laschi discloses the system of claim 16, wherein the one or more processors are configured to infer a grasp position for the first object of the one or more end effectors based on the visual information of the first object (see disclosure that after a learning phase, the system starts from visual data, calculates the position and orientation of the hand for grasping, selects the best-suited hand configuration, and predicts the tactile feedback after grasping, Abstract; see also disclosure of the evaluation of the neural network in controlling the grasping of the robot, Sections IV.A and IV.B, page 5, col. 1, last paragraph through page 6, col. 1).
Claim 23
With respect to claim 23, Laschi discloses the system of claim 22, wherein the one or more processors are configured to extract the information based on the inferred grasp position for the first object of the one or more end effectors (see disclosure that after a learning phase, the system starts from visual data, calculates the position and orientation of the hand for grasping, selects the best-suited hand configuration, and predicts the tactile feedback after grasping, Abstract; see also disclosure of the evaluation of the neural network in controlling the grasping of the robot, Sections IV.A and IV.B, page 5, col. 1, last paragraph through page 6, col. 1).
Claim 24
With respect to claim 24, Laschi discloses the system of claim 16, wherein the visual information of the first object includes an image of the first object, and the visual information of the second object includes an image of the second object (see disclosure that the Vision Module receives binocular images of the scene acquired by cameras in the robot head and provides information about geometric features of the object of interest, such as object shape, dimensions, position, and orientation, Section II, page 2, col. 1, second paragraph).
Claim 25
With respect to claim 25, Laschi discloses the system of claim 16, wherein the neural network model is trained using the tactile information of the second object acquired by one or more tactile sensors (see disclosure of the use of a humanoid robot that includes a hand with tactile sensors for grasping tasks, Section II, page 2, col. 1, first paragraph; see also disclosure that during a training phase, the system grasps different kinds of objects in different positions to learn correlations between visual information, hand and arm configurations, and tactile images, Section II, page 2, col. 1, last full paragraph).
Claim 29
With respect to claim 29, Laschi discloses the system of claim 16, wherein the one or more processors are configured to update at least one of a grasp position for the first object of the one or more end effectors or a grasp force for the first object of the one or more end effectors (see disclosure that after a learning phase, the system starts from visual data, calculates the position and orientation of the hand for grasping, selects the best-suited hand configuration, and predicts the tactile feedback after grasping, Abstract; see also disclosure of the evaluation of the neural network in controlling the grasping of the robot, Sections IV.A and IV.B, page 5, col. 1, last paragraph through page 6, col. 1).
Claim 30
With respect to claim 30, Laschi discloses the system of claim 29, wherein the one or more processors are configured to update the at least one of the grasp position or the grasp force based on reinforce learning (see disclosure that during a training phase, the system grasps different kinds of objects in different positions to learn correlations between visual information, hand and arm configurations, and tactile images, Section II, page 2, col. 1, last full paragraph).
Claim 31
With respect to claim 31, Laschi discloses the system of claim 16, wherein the one or more end effectors are configured to grasp the first object based on the extracted information (see disclosure that after a learning phase, the system starts from visual data, calculates the position and orientation of the hand for grasping, selects the best-suited hand configuration, and predicts the tactile feedback after grasping, Abstract; see also disclosure of the evaluation of the neural network in controlling the grasping of the robot, Sections IV.A and IV.B, page 5, col. 1, last paragraph through page 6, col. 1).
Claim 32
With respect to claim 32, Laschi discloses the system of claim 16, wherein the one or more processors are configured to control a grasp force of the one or more end effectors based on the extracted information (see disclosure that after a learning phase, the system starts from visual data, calculates the position and orientation of the hand for grasping, selects the best-suited hand configuration, and predicts the tactile feedback after grasping, Abstract; see also disclosure of the evaluation of the neural network in controlling the grasping of the robot, Sections IV.A and IV.B, page 5, col. 1, last paragraph through page 6, col. 1).
Claim 34
With respect to claim 34, Laschi discloses a method as claimed, comprising:
a) inputting, by one or more processors (inherent in a computerized robotic control system), at least visual information of a first object into a neural network model (see disclosure that the Tactile Prediction Module receives as input the geometric features of the object provided by the Vision Module, Section III.D, page 4, col. 1), and extracting, based on the neural network model, information to control one or more end effectors (see disclosure that the Tactile Prediction Module receives information about the object geometric features from the Vision Module and information from the Preshaping Module to the hand/arm configuration, and based on this information, the Tactile Prediction Module provides as output the tactile image expected when the object is contacted, Section II, page 2, col. 1, fourth paragraph); and
b) controlling, by the one or more processors (inherent in a computerized robotic control system), the one or more end effectors to manipulate the first object based on the extracted information (see disclosure that after a learning phase, the system starts from visual data, calculates the position and orientation of the hand for grasping, selects the best-suited hand configuration, and predicts the tactile feedback after grasping, Abstract; see also disclosure of the evaluation of the neural network in controlling the grasping of the robot, Sections IV.A and IV.B, page 5, col. 1, last paragraph through page 6, col. 1);
c) wherein the neural network model has been trained using visual information of a second object and tactile information of the second object (see disclosure that during training of the neural network, the system learns correlations between visual information, hand and arm configurations, and tactile images, Section II, page 2, col. 1, last full paragraph); and
d) wherein the visual information of the first object includes at least texture information of a surface of the first object, and the visual information of the second object includes at least texture information of the second object (see disclosure that during training the system learns correlations between visual information, hand and arm configurations, and tactile images, Section II, page 2, col. 1, last full paragraph; see also disclosure that the Tactile Prediction Module provides as output the tactile image expected when the object is contacted, Section II, page 2, col. 1, fourth paragraph; see also disclosure that the tactile image expected includes an estimation of object surface properties [i.e., texture information], Section II, first paragraph, page 1, col. 2).
XV. Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 2, 3, 17, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over “A Bio-inspired Neural Sensory-Motor Coordination Scheme for Robot Reaching and Preshaping” by Cecilia Laschi et al. (“Laschi”) as applied to claims 1 and 16 above, and further in view of U.S. Patent 11,004,191 to Massaru Adachi (“Adachi”).
Claims 2 and 17
Regarding claims 2 and 17, Laschi teaches the apparatus and system for estimating tactile information substantially as claimed, including utilizing a neural network model that has been trained to output the tactile information of the second object by inputting the visual information of the second object.
Laschi does not explicitly teach the apparatus and system wherein the neural network model is generated using an autoencoder.
Adachi, however, teaches a robot picker that utilizes a neural network model that has been trained using an autoencoder (see col. 6, lines 37-63).
It would have been obvious to a POSITA prior to the effective filing date of the application to utilize an autoencoder to train a neural network, because it would increase processing accuracy (see col. 6, lines 54-58).
Claims 3 and 18
Regarding claims 3 and 18, Laschi teaches the apparatus and system for estimating tactile information substantially as claimed, including utilizing a neural network model that has been trained to output the tactile information of the second object by inputting the visual information of the second object.
Laschi does not explicitly teach the apparatus and system wherein the neural network model is an encoder based on an autoencoder.
Adachi, however, teaches a robot picker that utilizes a neural network model is an encoder based on an autoencoder (see col. 6, lines 37-63).
It would have been obvious to a POSITA prior to the effective filing date of the application to utilize an autoencoder to train a neural network, because it would increase processing accuracy (see col. 6, lines 54-58).
Claims 9, 10, 20, and 27-30 are rejected under 35 U.S.C. 103 as being unpatentable over “A Bio-inspired Neural Sensory-Motor Coordination Scheme for Robot Reaching and Preshaping” by Cecilia Laschi et al. (“Laschi”) as applied to claims 1 and 16 above, and further in view of “Self-Supervised Regrasping using Spatio-Temporal Tactile Features and Reinforcement Learning” by Yevgen Chebotar et al. (“Chebotar”).
Claim 9
Regarding claim 9, Laschi teaches the apparatus substantially as claimed, including the training of the neural network model using tactile information of the second object.
Laschi does not explicitly teach the apparatus wherein the tactile information of the second object includes pressure information.
Chebotar, however, teaches the apparatus wherein the tactile information of the second object includes pressure information (see disclosure of the use of biomimetic tactile sensors4, or BioTacs, Section III.A, page 1961, col. 2, last paragraph).
It would have been obvious to a POSITA prior to the effective filing date of the application to utilize pressure information of the tactile sensors, since pressure information can be utilized to determine the grip pressure, or tightness, of the gripper against the object being gripped.
Claim 10
Regarding claim 10, Laschi teaches the apparatus substantially as claimed, including the training of the neural network model using tactile information of the second object.
Laschi does not explicitly teach the apparatus wherein the neural network model is trained using tactile information of the second object in time series.
Chebotar, however, teaches training the neural network model using tactile information of the second object in time series (see disclosure of the use of spatio-temporal feature descriptors for learning a sparse representation of the tactile sequence data for a grasp stability prediction, Section III, page 1961, col. 2, first full paragraph).
It would have been obvious to a POSITA prior to the effective filing date of the application to utilize tactile information in a time series, because this allows the achievement of high grasp stability prediction accuracy in a short time (Section II, page 1961, col. 2, first paragraph).
Claim 20
Regarding claim 20, Laschi teaches the system substantially as claimed, including the training of the neural network model using tactile information of the second object.
Laschi does not explicitly teach the system wherein the extracted information is a manipulation signal for the one or more end effectors.
Chebotar, however, teaches extracted information that is a manipulation signal for the one or more end effectors (see disclosure that the system uses reinforcement learning with spatio-temporal features which is supervised by the previously learned grasp stability predictor, which allows the learning of regrasping behavior, Section II, page 1961, col. 2, first paragraph).
It would have been obvious to a POSITA prior to the effective filing date of the application to utilize learned manipulation signals for the one or more end effectors, because this would allow the system to perform a regrasping behavior in an autonomous and efficient way (Section II, page 1961, col. 2, first paragraph).
Claim 27
Regarding claim 27, Laschi teaches the system substantially as claimed, including the input of visual information of the first object into the neural network model and the use of one or more tactile sensors.
Laschi does not explicitly teach the system wherein the one or more processors are configured to input the tactile information of the first object acquired by the one or more tactile sensors into the neural network model, and extract the information to control the one or more end effectors.
Chebotar, however, teaches inputting the tactile information of the first object acquired by the one or more tactile sensors into the neural network model, and extracting the information to control the one or more end effectors (see disclosure of a framework for learning regrasping behaviors based on tactile data, Abstract, using information acquired during an initial grasp of an object to regrasp the object, Section IV, page 1963, col. 1, first paragraph).
It would have been obvious to a POSITA prior to the effective filing date of the application to utilize tactile sensor information acquired during an initial grasp attempt to exert further control on the end effectors [i.e., regrasping], because the use of tactile information of the initial grasp attempt could be beneficially used to infer a local change of the gripper configuration that will improve the grasp stability in regrasping the object (Section I, page 1960, col. 1, third paragraph).
Claim 28
Regarding claim 28, Laschi teaches the system substantially as claimed, including the input of visual information of the first object into the neural network model and the use of one or more tactile sensors.
Laschi does not explicitly teach the system wherein the one or more processors are further configured to update manipulation of the first object based on tactile information of the first object acquired by the one or more tactile sensors.
Chebotar, however, teaches updating manipulation of the first object based on tactile information of the first object acquired by the one or more tactile sensors (see disclosure of a framework for learning regrasping behaviors based on tactile data, Abstract, using information acquired during an initial grasp of an object to regrasp the object, Section IV, page 1963, col. 1, first paragraph).
It would have been obvious to a POSITA prior to the effective filing date of the application to utilize tactile sensor information acquired during an initial grasp attempt to exert further control on the end effectors [i.e., regrasping], because the use of tactile information of the initial grasp attempt could be beneficially used to infer a local change of the gripper configuration that will improve the grasp stability in regrasping the object (Section I, page 1960, col. 1, third paragraph).
Claim 29
Regarding claim 29, to the extent that one could argue that Laschi fails to teach updating at least one of a grasp position for the first object of the one or more end effectors or a grasp force for the first object of the one or more end effectors, Chebotar teaches this limitation.
Specifically, Chebotar teaches updating at least one of a grasp position for the first object of the one or more end effectors or a grasp force for the first object of the one or more end effectors (see disclosure of a framework for learning regrasping behaviors based on tactile data, Abstract, using information acquired during an initial grasp of an object to regrasp the object, Section IV, page 1963, col. 1, first paragraph).
It would have been obvious to a POSITA prior to the effective filing date of the application to utilize tactile sensor information acquired during an initial grasp attempt to exert further control on the end effectors [i.e., regrasping], because the use of tactile information of the initial grasp attempt could be beneficially used to infer a local change of the gripper configuration that will improve the grasp stability in regrasping the object (Section I, page 1960, col. 1, third paragraph).
Claim 30
Regarding claim 30, to the extent that one could argue that Laschi fails to teach updating at least one of the grasp position or the grasp force based on reinforce information, Chebotar teaches this limitation.
Specifically, Chebotar teaches updating at least one of a grasp position for the first object of the one or more end effectors or a grasp force for the first object of the one or more end effectors (see disclosure of a framework for learning regrasping behaviors based on tactile data, Abstract, using information acquired during an initial grasp of an object to regrasp the object, Section IV, page 1963, col. 1, first paragraph) based on reinforce[ment] learning (see title, Section IV, page 1963).
It would have been obvious to a POSITA prior to the effective filing date of the application to utilize tactile sensor information acquired during an initial grasp attempt to exert further control on the end effectors [i.e., regrasping], because the use of tactile information of the initial grasp attempt could be beneficially used to infer a local change of the gripper configuration that will improve the grasp stability in regrasping the object (Section I, page 1960, col. 1, third paragraph).
It would have been obvious to a POSITA prior to the effective filing date of the application to utilize reinforcement learning because this approach has enjoyed success in many different applications including grasping tasks, where it is used for determining where and how to grasp an unknown object, Section II, page 1961, col. 1, second full paragraph).
Claims 12 and 26 are rejected under 35 U.S.C. 103 as being unpatentable over “A Bio-inspired Neural Sensory-Motor Coordination Scheme for Robot Reaching and Preshaping” by Cecilia Laschi et al. (“Laschi”) as applied to claims 1 and 16 above, and further in view of “A Hybrid Deep Architecture for Robotic Grasp Detection” by Di Guo et al. (“Guo”).
Claims 12 and 26
Regarding claims 12 and 26, Laschi teaches the apparatus and system substantially as claimed, including the use of visual information of the first and second object.
Laschi does not explicitly teach the apparatus and system wherein the visual information of the first object includes depth information and the visual information of the second object includes depth information.
Guo, however, teaches the use of visual information that includes depth information (see disclosure of the use of depth information, Section V.A, page 1612, col. 1, last paragraph).
It would have been obvious to a POSITA prior to the effective filing date of the application to utilize depth information of the first and second objects since this could be used to segment the respective object from the background, Section V.A, page 1612, col. 1, last paragraph.
Claim 33 is rejected under 35 U.S.C. 103 as being unpatentable over “A Bio-inspired Neural Sensory-Motor Coordination Scheme for Robot Reaching and Preshaping” by Cecilia Laschi et al. (“Laschi”) as applied to claim 16 above, and further in view of U.S. Patent 10,792,809 to Jeffrey Bingham et al. (“Bingham”).
Claim 33
Regarding claim 33, Laschi teaches the system substantially as claimed, including the use of one or more end effectors.
Laschi does not explicitly teach the system wherein the one or more end effectors are provided with the one of more visual sensors.
Bingham, however, teaches the use of one or more end effectors provided with one or more visual sensors (see disclosure of the use of an infrared camera on a robotic gripping device, col. 1, lines 36-46).
It would have been obvious to a POSITA prior to the effective filing date of the application to utilize a visual sensor on the end effector, since certain tasks may be difficult to perform with only data from remote sensors, such as a head-mounted camera, and so it may be advantageous to position one or more sensors on or proximate to a robotic gripper or other end effector of a robot, col. 4, line 58 through col. 5, line 3.
XVI. Conclusion
In accordance with MPEP § 1406, the examiner has reviewed and considered the prior art cited or of record in the original prosecution of the ‘658 patent. Applicants are reminded that a listing of the information cited or of record in the original prosecution of the ‘658 patent need not be resubmitted in this reissue application unless Applicant(s) desire the information to be printed on a patent issuing from this reissue application.
Applicant(s) are reminded of the continuing obligation under 37 CFR § 1.178(b), to timely apprise the Office of any prior or concurrent proceeding in which ‘658 patent is or was involved. These proceedings would include interferences, reissues, reexaminations, other post-grant proceedings in the Office, and litigation.
Applicant(s) are further reminded of the continuing obligation under 37 C.F.R. § 1.56, to timely apprise the Office of any information which is material to patentability of the claims under consideration in this reissue application.
These obligations rest with each individual associated with the filing and prosecution of this application for reissue. See also MPEP §§ 1404, 1442.01 and 1442.04.
Applicant(s) are also reminded that any amendments to the claims must comply with the provisions of 35 U.S.C. § 112 first paragraph, having clear support and antecedent basis in the specification. See 37 C.F.R. § 1.75(d)(1) and MPEP § 608.01(o).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Luke S. Wassum whose telephone number is (571) 272-4119. The examiner can normally be reached on Monday - Friday 8 AM-5 PM, alternate Fridays off.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Fuelling can be reached on 571-270-1367. The fax phone number for the organization where this application or proceeding is assigned is 571-273-9900.
In addition, INFORMAL or DRAFT communications may be faxed directly to the examiner at 571-273-4119. Such communications must be clearly marked as INFORMAL, DRAFT or UNOFFICIAL.
Patent Center
Patent Center is available to all users for electronic filing and management of patent applications. For more information, please visit the Patent Center information page at www.uspto.gov/patents/apply/patent-center.
/LUKE S WASSUM/Primary Examiner, Art Unit 3992
Conferees:
/Stephen J. Ralis/Primary Examiner, Art Unit 3992 Michael Fuelling /MF/
Supervisory Patent Examiner
Art Unit 3992
lsw
17 August 2026
1 Eureka Blog "The Ultimate Guide to End Effector: Everything You Need to Know", downloaded from https://eureka.patsnap.com/blog/machinery-tech-resources/what-is-an-end-effector/, 31 July 2024.
2 Ferrobotics "What is an End Effector?", downloaded from https://www.ferrobotics.com/en/news/what-is-an-end-effector-and-or-end-of-arm-tool-eoat/, 2026.
3 See, e.g., Ariad Pharmaceuticals, Inc. v. Eli Lilly & Co., 598 F.3d 1336, 1341 (Fed. Cir. 2010) (en banc).
4 Biomimetic tactile sensors (BioTacs) are described in Wettel et al. “Biomimetic Tactile Sensor Array”, 2008, which discloses that one of the outputs of the sensor is fluid pressure.
Prosecution Insights