DETAILED ACTION
Remarks
This non-final office action is in response to the application filled on 09/15/2023. Claims 1-20 are pending and examined below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a) ‐ (d). The certified copy has been filed in parent Application No. EP 2017/21163107.2, filed on 03/17/2021.
Information Disclosure Statement
As of date of this action, IDS filled has been annotated and considered.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
“device” in claim 14, 15 and 16
“Actuation means” in claim 15
[0012] of PGPub of submitted specification describe device comprising a robot element. Submitted specification does not describe structure and configuration of actuation means.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION. —The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Regarding claim 1 (and similarly claim 15), which recites “computing the technical instruction…based on the segmented voxel representation and the measurement” is not clear. It is not clear whether instruction is computed based on first segment voxel or second segment voxel or both.
Dependent claim(s) 2-14 and 16-20 is/are also rejected because they do not resolve their parent deficiencies.
Regarding claim 14, which recites “a three dimensional… physical object”. From the recited claim, it is not clear if “a three dimensional… physical object”, is the same object as claim 1 or not. Examiner suggests that Applicant make Claim 14 separate and incorporate the text of Claim 1 into it instead of referring to Claim 1 in order to avoid antecedent issues.
Regarding claim 19, which recites “said segmented representation” is not clear. It is not clear whether said segmented representation is referring first segment or second segment or both.
Claim 15 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim limitation “actuation means” invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function. Submitted specification does not describe structure and configuration of actuation means. It is not clear whether a motor is acting as actuation means or something else. Therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
Applicant may:
(a) Amend the claim so that the claim limitation will no longer be interpreted as a limitation under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph;
(b) Amend the written description of the specification such that it expressly recites what structure, material, or acts perform the entire claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(c) Amend the written description of the specification such that it clearly links the structure, material, or acts disclosed therein to the function recited in the claim, without introducing any new matter (35 U.S.C. 132(a)).
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts and clearly links them to the function so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed function, applicant should clarify the record by either:
(a) Amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(b) Stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed function. For more information, see 37 CFR 1.75(d) and MPEP §§ 608.01(o) and 2181.
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claim 15 is rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
As described above, the specification lacks structure that is clearly linked to the limitations interpreted under 112(f) of claim limitation “actuation means” of claim 15. Submitted specification does not describe structure and configuration of actuation means.
Because there is no disclosure of adequate structure to perform the claimed function, the specification does not convey with reasonable clarity to those skilled in the art that the applicant had possession of the claimed invention.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 6, 7, 14-16 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over IEEE/RSJ International conference on intelligent robots and systems (IROS), title “3D object classification with point convolution network”, by (“Chen”), submitted as IDS on 09/15/2023, and further in view of US 2022/0284544 (“Holzer”).
Regarding claim 1, as best understood in view of indefiniteness rejection explained above, Chen discloses a method for generating a technical instruction for handling a three-dimensional physical object present within a reference volume, the physical object comprising a 3D surface (see at least page 783, left col, where “Recognizing objects from the point cloud captured by modern 3D sensors is an important task for robots operating autonomously in real-world environments.”; see also section I and fig 1, where 3D surfaces of objects are shown. The measurements and detection are done in a real environment which is within a defined volume. Defined real world environment is interpreted as reference volume), the method comprising:
- obtaining at least two images of the physical object from a plurality of (see at least abstract, where “modern 3D sensors”; see also fig 5 and page 784, section A- B);
- generating, with respect to the 3D surface of the physical object, a voxel representation segmented based on the at least two images (see at least page 785), said segmenting comprising identifying a first segment component corresponding to a plurality of first voxels and a second segment component corresponding to a plurality of second voxels different from the plurality of first voxels (see at least page 785, section C);
- performing a measurement with respect to the plurality of first voxels (see at least page 784, section III, A); and
- computing the technical instruction, said technical instruction preferably comprising a robot command, for the handling of the physical object based on the segmented voxel representation and the measurement (see at least page 783, section I, where “robot grasping”),
wherein said segmenting relates to at least one trained neural network (NN) being trained with respect to the 3D surface (see at least page 783, section I).
Chen does not disclose the following limitation:
a plurality of cameras positioned at different respective angles with respect to the physical object.
However, Holzer discloses method wherein a plurality of cameras positioned at different respective angles with respect to the physical object (see at least [0149], where “a damage detection portal may have multiple cameras that capture image data of the object at different angles and from different viewpoints.”).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Chen to incorporate the teachings of Holzer by including the above feature for providing more accurate object detection by capturing images at different angles.
Regarding claim 2, Chen further discloses a method wherein the technical instruction comprises a robot command, wherein the robot command is executable by means of a device comprising a robot element configured for handling the physical object (see at least page 783, section I, where “robot grasping”).
Regarding claim 6, Chen further discloses a method wherein the generating comprises:
- performing a 3D reconstruction of the 3D surface of the physical object based on the at least two images for obtaining a voxel representation (see at least page 783, section I and page 784, section III.B), and
- obtaining said segmented voxel representation by projecting at least the first segment component with respect to said voxel representation (see at least page 784, section III.A),
wherein the at least one trained NN comprises an instance segmentation NN, being a two-dimensional (2D) and/or 3D region-based convolutional neural network (R-CNN) or being a Mask R-CNN for segmenting the at least two images and/or a 3D-BoNet for segmenting the voxel representation (see at least page 783, section I and page 787, section V).
Regarding claim 7, Chen further discloses a method wherein the generating comprises:
- performing a 3D reconstruction of the 3D surface of the physical object based on the at least two images for obtaining a voxel representation (see at least page 783, section I), and
- obtaining said segmented voxel representation by projecting at least the first segment component with respect to said voxel representation (see at least page 784, section III.A),
wherein the at least one trained NN comprises a semantic segmentation NN, being a 2D and/or 3D convolutional neural network, CNN, or being a 2D U-net for segmenting the at least two images and/or a PointNet++ for segmenting the voxel representation (see at least page 783, section I).
Regarding claim 14, Chen in view of Holzer further discloses a device for handling a three-dimensional, 3D, physical object present within a reference volume, the physical object comprising a 3D surface, the device comprising a robot element, a processor and memory comprising instructions which, when executed by the processor, cause the device to execute a method according claim 1 (see citation on claim 1).
Regarding claim 15, as best understood in view of indefiniteness rejection explained above, Chen in view of Holzer further discloses claim 15 (see citation on claim 1 and 2).
Regarding claim 16, Chen in view of Holzer further discloses a non-transitory computer readable medium containing a computer executable software which when executed on a device, performs the method of claim 1 (the method of Chen is perform on a computer. see citation on claim 1).
Regarding claim 19, as best understood in view of indefiniteness rejection explained above, Chen further discloses a method wherein obtaining of said segmented voxel representation comprises performing clustering with respect to a projected at least first segment component (see at least page 784, section III, where “The input of point convolution network consists of the set of normals of the segmented point cloud. Fig. 1 shows the point cloud and its normals.”; see also page 783, section II).
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen and Holzer, as applied to claim 1 above, and further in view of US 2020/0211206 (“Wang”).
Regarding claim 3, Chen in view of Holzer does not disclose claim 3. However, Wang discloses a method comprising:
- pre-processing of the at least two images based on a mask projection for distinguishing foreground from background, said mask projection being based at least partially on a mask-related 3D reconstruction of the 3D surface of the physical object (see at least [0115], where “The moving object mask is provided as a binary map to distinguish between static background and moving foreground in flow evaluation.”; see also [0025] and [0149]).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Chen in view of Holzer to incorporate the teachings of Wang by including the above feature for determining accurate geometry and 3D motion shape model of the object.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen and Holzer, as applied to claim 1 above, and further in view of US 2001/0005204 (“Matsumoto”).
Regarding claim 4, Chen in view of Holzer does not disclose claim 4. However, Matsumoto discloses a method wherein said segmenting comprises identifying a third segment component corresponding to a plurality of third voxels comprised in the plurality of first voxels, wherein at least one of the first voxels does not belong to the third voxels (see at least [0374], where “an object image that is the average of the plurality of object images is segmented into a plurality of regions R at step S551.”; see also [0389], where “Referring to FIG. 36, cylindrical coordinate system voxel space 251 includes a plurality of voxels 39.”),
wherein the measurement is performed with respect to the plurality of third voxels (see at least fig 31 and fig 36).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Chen in view of Holzer to incorporate the teachings of Matsumoto by including the above feature for determining accurate shape model of the object.
Claim(s) 5 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen and Holzer, as applied to claim 1 above, and further in view of US 2018/0286117 (“Redden”).
Regarding claim 5, Chen in view of Holzer does not disclose claim 5. However, Redden discloses a method wherein the 3D surface of the physical object is a plant comprising, one or more leaves, corresponding to the first segment component (see at least [0013-18]).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Chen in view of Holzer to incorporate the teachings of Redden by including the above feature for providing wide variety of object recognition.
Regarding claim 18, Redden further discloses a method wherein the plant comprises soil and/or one or more roots, corresponding to the second segment component (see at least fig 1A, where skeleton segment include ground plane).
Claim(s) 8, 9 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen and Holzer, as applied to claim 1 above, and further in view of ARXIV.org, Cornell university library, title: “Vision-based Robotic Grasping from Object Localization, Object Pose Estimation to Grasp Estimation for Parallel Grippers: A Review” by (“Du”), submitted as IDS.
Regarding claim 8, Chen in view of Holzer does not disclose claim 8. However, Du discloses a method wherein the measurement relates to counting with respect to the segmented voxel representation, and wherein the segmented voxel representation is obtained via semantic segmentation for counting clusters of voxels and/or instance segmentation for counting instances (see at least page 9, section 2.3).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Chen in view of Holzer to incorporate the teachings of Du by including the above feature for providing slip and damage free grasp during object handling by localizing object accurately by counting clusters of voxels.
Regarding claim 9, Chen in view of Holzer does not disclose claim 9. However, Du further discloses a method wherein the measurement comprises determining any one or combination of: a number of elements, an area and a volume of said segment component based on counting with respect to the segmented voxel representation (see at least page 9, section 2.3 and fig 9).
Regarding claim 20, Chen in view of Holzer does not disclose claim 20. However, Du further discloses a method wherein the measurement comprises determining any one or combination of: a height and an angle of said segment component with respect to a main direction comprised in the reference volume based on counting of a plurality of voxels associated with said segment component along the main direction representation (see at least page 9, section 2.3 and fig 15).
Claim(s) 10 and 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen and Holzer, as applied to claim 1 above, and further in view of US 2021/0122586 (“Sun”).
Regarding claim 10, Chen further discloses a method wherein the handling comprises (see at least fig 7).
Chen in view of Holz does not disclose the following limitation:
physically sorting the physical object according to respective physical destination locations corresponding to respective classes.
However, Sun discloses a method wherein physically sorting the physical object according to respective physical destination locations corresponding to respective classes (see at least [0045], where “use the routing information to sort the item to a corresponding destination, such as a pile, bin, or other set of items destined for the same next intermediate and/or final destination.”).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Chen in view of Holzer to incorporate the teachings of Sun by including the above feature for increasing efficiency by placing object based on final destination.
Regarding claim 11, Chen in view of Holzer does not disclose claim 12. However, Sun further discloses a method wherein the handling comprises physically separating a sample from the physical object at a handling coordinate based on said measurement (see at least fig 2A).
Claim(s) 12 and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen and Holzer, as applied to claim 1 above, and further in view of US 2022/0072707 (“Fan”).
Regarding claim 12, Chen in view of Holzer does not disclose claim 12. However, Fan discloses a method wherein a 3D approaching angle for reaching the handling coordinate on said physical object relates to a 3D sampling angle for separating the sample at the handling coordinate (see at least [0059], [0061] and fig 4).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Chen in view of Holzer to incorporate the teachings of Fan by including the above feature for generating high quality grasp candidates without teaching manually.
Regarding claim 13, Fan further discloses a method comprising:
-actuating a robot element based on a robot command, wherein said actuating comprises (see at least fig 4):
-approaching, by the robot element, the 3D surface at a 3D approaching angle (see at least [0024] and [0061]); and
-separating, by the robot element, the sample from the physical object at the handling coordinate (see at least fig 4),
wherein the step of separating comprises surrounding, by two distal ends of the robot element, a receiving portion of the physical object at the 3D sampling angle, wherein the 3D sampling angle relates to an orientation of the two distal ends of the robot element with respect to a main plane of the receiving portion (see at least fig 4 and [0046]).
Claim(s) 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen and Holzer, as applied to claim 1 above, and in view of US 2001/0005204 (“Matsumoto”), as applied to claim 4 above, and further in view of US 2018/0286117 (“Redden”).
Regarding claim 17, Chen, Holzer and Matsumoto does not disclose claim 17. However, Redden further discloses a method wherein one or more stems corresponding to the third segment component (see at least fig 1A). Same motivation of claim 5 applies.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SOHANA TANJU KHAYER whose telephone number is (408)918-7597. The examiner can normally be reached on Monday - Thursday, 7 am-5.30 pm, PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abby Lin can be reached on 571-270-3976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SOHANA TANJU KHAYER/Primary Examiner, Art Unit 3657