DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant submitted a Reply on 05 March 2026 that, inter alia:
Cancelled claims 12-14 thus overcoming the 112(a) rejection of those claims but also adding new claims 27-28 are similar to claim 14 and subject to the 112(a) rejection;
Cancelled claim 18 thus overcoming the 112(b) rejection of that claim but also adding new claim 29 which is similar to claim 18 and subject to the 112(b) rejection; and
Added new claims 21-34 but which are rejected on prior art below.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 27 and 28 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
The instant specification fails to adequately disclose (claim 27) wherein the artificial intelligence (AI) system is configured to determine paths of movement of the robot that avoid singularities where the robot is incapable of moving or (claim 28) wherein the artificial intelligence (AI) system is configured to determine paths of movement of the robot that avoid singularities that cause mathematical errors in the artificial intelligence (AI) system.
Although [0253] defines the term “singularity” the specification fails to disclose how to prevent singularities by determining movement paths that avoid such singularities. Furthermore, it is not understood how this purported feature relates to or involves the modified images for training the AI system. Moreover, there is no reference to conventional techniques for such functions such that the specification purports to disclose and claim these functionalities with a single sentence wholly lacking necessary technical details such as apparatus elements and methodology necessary to reasonably convey to one skilled in the relevant art that the inventors, at the time the application was filed, had possession of the claimed invention.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 22-25 and 29-34 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “human imperceptible image qualities, and the human imperceptible image qualities are imperceptible to humans” in claims 22-25 and 29-34 is a relative term which renders the claim indefinite. The term “human imperceptible” is not defined by the claim, the specification does not provide an adequate standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. Moreover, human beings vary widely in their physical and abilities to distinguish between images and perceive image qualities such that images that appear identical to Bob may appear to be different images having perceptibly different image qualities to Sally. Still further, Sally may have extensive training and knowledge regarding various image capturing and processing techniques while Bob may be a complete novice.
It is recognized that the term in question is discussed in various portions of the specification but this term remains too nebulous and ill-defined such that it also fails the infringement test in that a competitor would not be reasonably apprised of the claim scope. In other words, this phrase fails to serve the notice function required by 35 U.S.C. 112(b) by not providing clear warning to others as to what constitutes infringement of the patent. See, e.g., Solomon v. Kimberly-Clark Corp., 216 F.3d 1372, 1379, 55 USPQ2d 1279, 1283 (Fed. Cir. 2000). See also In re Larsen, 10 Fed. App'x 890 (Fed. Cir. 2001) and MPEP 2173.02(II).
Response to Arguments
Applicant’s arguments with respect to claims 1 and 15 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1, 15, 20, 22, 23, 25, and 27-30 are rejected under 35 U.S.C. 103 as being unpatentable over Anaya {Anaya, Josue, and Adrian Barbu. "Renoir–a dataset for real low-light image noise reduction." Journal of Visual Communication and Image Representation 51 (2018): 144-154}, Horváth {D. Horváth, G. Erdős, Z. Istenes, T. Horváth and S. Földi, "Object Detection Using Sim2Real Domain Randomization for Robotic Applications," in IEEE Transactions on Robotics, vol. 39, no. 2, pp. 1225-1243, April 2023, doi: 10.1109/TRO.2022.3207619), and Nanni {Nanni, Loris, et al. "Comparison of different image data augmentation approaches." Journal of imaging 7.12 (2021): 254}.
Claim 1
In regards to claim 1, Anaya discloses a system, comprising:
a camera configured to capture two or more images of one or more objects that are stationary
{see section 1, two digital cameras are used to capture a dataset of images of an object using cameras of different sensor sizes and different noise levels while section 2 and pg. 2 clarifies that the dataset is of a static (stationary) scene};
wherein the camera is mounted
wherein the camera is mounted at a fixed position to form a static field of view with a same viewing angle when the images are captured {see section 2 and pg. 6 capturing images from a static scene by not moving the camera thus indicating that the camera has the same viewing angle when the images are captured, and pg. 8 tripod mounting of camera in fixed position};
wherein the objects and a scene surrounding the objects remain unchanged as the camera captures the images {see section 2 and pg. 6 capturing images from a static scene by not moving the camera thus indicating that the objects within the scene remain unchanged as the camera captures the images};
an artificial intelligence (AI) system operatively coupled to the camera, wherein the images of the objects form an image training set that is used by the artificial intelligence (AI) system to develop a machine learning model for the objects
{see the noise reduction algorithm, section I and pgs. 3-4 stating that the image data set generated by the disclosure may be used for training image denoising algorithms such as AI systems trained by the dataset to develop machine learning image denoising models such as Active Random Field, Multi-Layer Perceptron to produce denoising results such as those illustrated in Table 3 and discussed in Section 5};
wherein the image training set has one or more modified images with at least one image property different from the unmodified images, wherein the camera is configured to create the modified images by changing the image property, artificial Intelligence (AI) system is configured to create the modified images by controlling the camera;
{the camera is controlled to change the light sensitivity (ISO) and exposure time to create an image training set having modified images, having image properties (ISO, exposure time) different than, for example, the reference image or clean images as per section 2.1};
wherein the modified images are at most
Although Anaya does not explicitly disclose a robot the disclosed modified image data training set and AI model are clearly adapted to various robotic applications because removing or reducing noise improves downstream operations performed by the broadly recited robot, pg. 1 in which, in the computer vision field of which robots are a part, noise reduction is synonymous with improvement in image quality such that robot applications of such trained models include retail industry, medicine, industrial manufacturing and many others all of which would benefit from the enhanced image data training set generated by Anaya.
Also, many authors have widely recognized the motivation to apply data augmentation to DL models because they prevent overfitting of the model by increasing the diversity of the training set. See, for, example, Shorten, Connor, and Taghi M. Khoshgoftaar. "A survey on image data augmentation for deep learning." Journal of big data 6.1 (2019): 1-48.
Horváth is a highly analogous reference from the same field of machine learning and robotics and solves the same or similar problem of preventing overfitting of the machine learning model by using data augmentation. See below cites.
Horváth more specifically teaches, inter alia, the following elements:
a robot, wherein the camera is mounted proximal to the robot {see title, abstract, Sections IV(C), VI IX, X, Figs. 5, 7, 23-24 including a frame that holds (mounted) the camera proximal to the robot}
an artificial intelligence (AI) system operatively coupled to the camera, wherein the images of the objects form an image training set that is used by the artificial intelligence (AI) system to develop a machine learning model for the objects
{See section IX Robotic Application, Figs. 23, 24 (copied below) including a machine learning model (computer vision module) trained in object detection and used for various robotic application such as pick-and-place applications wherein the camera is mounted proximal to the robot}.
PNG
media_image1.png
368
620
media_image1.png
Greyscale
};
wherein the image training set has one or more modified images with at least one image property different from the other images {See sections IVC and Table V teaching various data augmentation tools used to modify images that are then used in the training process of the machine learning model};
a robot configured to handle one or more items based on the machine learning model developed by the artificial intelligence (AI) system
{see section IX Robotic Application, Figs. 23, 24 including a machine learning model trained in object detection and used for various robotic application such as pick-and-place applications using a six-DoF robot arm equipped with camera, force sensor and gripper that is configured to handle items}.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Anaya which already discloses creating modified images of an object using a camera mounted at a fixed position to form a static field of view with the same viewing angle in which the modified images form an image training set for machine learning model AI system and which is adapted to various robotic applications including robots using computer vision which would benefit from being trained by modified images of the image training set such that the system explicitly includes a robot configured to handle one or more items based on the machine learning model developed by the artificial intelligence (AI) system, wherein the camera is mounted proximal to the robot as taught by Horvath because such models heavily rely on comprehensive data sets and that increasing the diversity of the training data and using real-world images modified by controlling the camera improves performance as motivate by Anaya in the abstract, introduction, section 4.4, Table 3, and Section 5, because there is a reasonable expectation of success, and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Anaya teaches wherein the modified images are at most
Nanni is an analogous reference from the same field of machine learning and solves the same or similar problem of preventing overfitting of the machine learning model by using data augmentation. See abstract, Introduction.
Nanni also teaches that the modified images created by augmentation may form a variety of percentages in the training set including wherein the modified images are at most 20% of the image training set See Table 1 copied below.
PNG
media_image2.png
470
978
media_image2.png
Greyscale
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Anaya which already discloses data augmentation techniques and AI models are clearly adapted to various robotic applications such that the system employs image training sets wherein the modified images are at most
Claims 27-28
Anaya is not relied upon to disclose (claim 27) wherein the artificial intelligence (AI) system is configured to determine paths of movement of the robot that avoid singularities where the robot is incapable of moving or (claim 28) wherein the artificial intelligence (AI) system is configured to determine paths of movement of the robot that avoid singularities that cause mathematical errors in the artificial intelligence (AI) system.
Initially, see the 112(a) rejection above in which each of these claimed camera functions has not been sufficiently disclosed. Moreover, this same claim language quite broadly refers to generalized and conventional abilities of robotic pick-and-place systems and without having any claimed link or relationship between the machine learning component of claim 1 and the recited, generalized robotic functions.
As to prior art mapping, see Horvath section IX Robotic Application teaching a robotic pick-and-place robot performing, as part of the operations, object detection using the machine learning computer vision model enhanced by augmented image training data. The computer vision model successfully localized, classified objects and physically grasped the objects which includes estimating the grasping pose and modifying the picking process to localize and grasp thus satisfying the broadly claimed functions of claims 12-14. Moreover, this use case was also presented in an exhibition thus also apparently demonstrating (claims 27 and 28), wherein the camera is configured to interface with the AI system to limit movement of the robot to prevent singularities including successfully determining paths of movement of the robot that avoid singularities where the robot is incapable of moving and successfully during the exhibiting determining paths of movement of the robot that avoid singularities that cause mathematical errors in the artificial intelligence (AI) system.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified the base combination including Anaya and Nesteruk which already discloses data augmentation techniques and AI models that are clearly adapted to various robotic applications including digital agriculture discussed in Nesteruk’s sections I, IIA, and IV which include plant condition sensing, spoiling detection, robotic weed control as well as other robot applications such as retail industry, medicine, industrial manufacturing and many others all of which would benefit from data-augmentation-enhanced-training of AI machine learning models used by robots to handle items such as picking out rotten fruit from a conveyor or robotic weed control of identified weed objects such that the system explicitly includes a robot configured to handle one or more items based on the machine learning model developed by the artificial intelligence (AI) system as taught by Horvath and further such that (claim 27) wherein the artificial intelligence (AI) system is configured to determine paths of movement of the robot that avoid singularities where the robot is incapable of moving or (claim 28) wherein the artificial intelligence (AI) system is configured to determine paths of movement of the robot that avoid singularities that cause mathematical errors in the artificial intelligence (AI) system as also taught by Horvath because Nesteruk motivates the use of data augmentation for DL models for robotic applications such as pick-and-place functionality; because Horvath motivates doing so because the disclosed Horvath computer vision model successfully localized, classified objects and physically grasped the objects which includes estimating the grasping pose and modifying the picking process to localize and grasp thus satisfying the broadly claimed functions of claims 12-14; because such models heavily rely on comprehensive data sets and that increasing the diversity of the training data improves model generalization, because there is a reasonable expectation of success, and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Independent Claim 15
In regards to claim 15, Anaya discloses a method, comprising:
mounting a camera at a fixed position to form a static field of view with a same viewing angle: capturing with the camera
{see section 1, two digital cameras are used to capture a dataset of images of an object using cameras of different sensor sizes and different noise levels while section 2 and pg. 2 clarifies that the dataset is of a static (stationary) scene and that 4 images per stationary scene are captured. See also pg. 8 tripod mount for the camera}
creating an image training set from the images of the objects
{see the noise reduction algorithm, section I and pgs. 3-4 stating that the image data training set generated by the disclosure may be used for training image denoising algorithms such as AI systems trained by the dataset to develop machine learning image denoising models such as Active Random Field, Multi-Layer Perceptron to produce denoising results such as those illustrated in Table 3 and discussed in Section 5 } and
modifying
{the camera is controlled to change the light sensitivity (ISO) and exposure time to create an image training set having modified images, having image properties (ISO, exposure time) different than, for example, the reference image or clean images as per section 2.1. According to 2.1 two reference/clean images and one to two modified (noisy) images are captured to form the dataset such that (1 to 2)/4 = 25% to 50% of the training set is modified};
developing with a machine learning system a machine learning model for the objects based at least on the image training set that contains modified images
{see the noise reduction algorithm, section I and pgs. 3-4 stating that the image data set generated by the disclosure may be used for training image denoising algorithms such as AI systems trained by the dataset to develop machine learning image denoising models such as Active Random Field, Multi-Layer Perceptron to produce denoising results such as those illustrated in Table 3 and discussed in Section 5} }; and
Horváth is a highly analogous reference from the same field of machine learning and robotics and solves the same or similar problem of preventing overfitting of the machine learning model by using data augmentation. See below cites.
Horváth more specifically teaches, inter alia, the following elements:
a robot, wherein the camera is mounted proximal to the robot {see title, abstract, Sections IV(C), VI IX, X, Figs. 5, 7, 23-24 including a frame that holds (mounted) the camera proximal to the robot}
an artificial intelligence (AI) system operatively coupled to the camera, wherein the images of the objects form an image training set that is used by the artificial intelligence (AI) system to develop a machine learning model for the objects and including handling items in a warehouse facility with a robot based on the machine learning model.
{See section IX Robotic Application, Figs. 23, 24 (copied below) including a machine learning model (computer vision module) trained in object detection and used for various robotic application such as pick-and-place applications wherein the camera is mounted proximal to the robot. Furthermore, pick-and-place applications include a variety of workplaces including, a broadly recited, a warehouse facility (e.g. building) with a robot handling items using pick-and-place methods that are based on the disclosed object detection machine learning model. See also {see section IX Robotic Application, Figs. 23, 24 including a machine learning model trained in object detection and used for various robotic application such as pick-and-place applications using a six-DoF robot arm equipped with camera, force sensor and gripper that is configured to handle items. };
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Anaya which already discloses creating modified images of an object using a camera mounted at a fixed position to form a static field of view with the same viewing angle in which the modified images form an image training set for machine learning model AI system and which is adapted to various robotic applications including robots using computer vision which would benefit from being trained by modified images of the image training set such that the system explicitly includes a robot and handling items in a warehouse facility with the robot based on the machine learning model as taught by Horvath because such models heavily rely on comprehensive data sets and that increasing the diversity of the training data and using real-world images modified by controlling the camera improves performance as motivate by Anaya in the abstract, introduction, section 4.4, Table 3, and Section 5, because there is a reasonable expectation of success, and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Anaya wherein the modified images are at most
Nanni is an analogous reference from the same field of machine learning and solves the same or similar problem of preventing overfitting of the machine learning model by using data augmentation. See abstract, Introduction.
Nanni also teaches that the modified images created by augmentation may form a variety of percentages in the training set including wherein the modified images are at most 20% of the image training set and generating training sets with ten or more images of one or more objects that have been captured with a camera. See Table 1 and Section 3 Materials and Methods.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Anaya which already discloses data augmentation techniques and AI models are clearly adapted to various robotic applications such that the system employs image training sets wherein the modified images are at most 20% of the image training set as taught by Nanni and to generates training sets with ten or more images of one or more objects as also taught by Nanni such that Anaya’s image capturing process that generates the training images captures ten or more images of object with the camera because there is a reasonable expectation of success, and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 16 Canceled
Claim 18 Canceled
Claim 20
In regards to claim 20, Anaya discloses wherein the objects are stationary when the camera captures the images {see mapping of claims 1, 15 in which the objects are stationary during/when the camera captures the images).
Claims 22-23, 29 and 30
Anaya discloses (claim 22) wherein any differences between the images forming the image training set before the modifying are from one or more human imperceptible image qualities, and the human imperceptible image qualities are imperceptible to humans and (claim 23) wherein the human imperceptible image qualities include variations in an index of refraction of air captured by the camera {see the 112b rejection of claim 22; (claim 29) wherein any differences between the images forming the image training set before modification are from one or more human imperceptible image qualities, and the human imperceptible image qualities are imperceptible to humans; and (claim 30) wherein the human imperceptible image qualities include heat vapor mirages The images captured by Anaya’s camera inherently include “human imperceptible image qualities such as variations in the index of refraction of air” and “heat vapor” images because such variations are caused by naturally occurring temperature variations in the air and/or objects being imaged. Indeed, these variations are so small they cannot be perceived by a human; moreover, the camera is not claimed or disclosed as having any particular or special property or function that would create such variations or heat vapor or otherwise capture such variations such that the cameras used by Anaya would also capture these imperceptible variations and naturally occurring heat vapor images.
Claim 25
In regards to claim 25, Anaya wherein the human imperceptible image qualities include stray pixel noise in the camera {Anaya’s cameras also suffer from conventional noise within the image sensor and the disclosed techniques are designed to enhance or exploit this noise using low light conditions as per the mapping of claims 1 and 15. See also [0202] of the instant spec in which the camera 205 is conventional and disclosed as having noise within the image sensor of the camera}.
Claims 4, 7, 17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Anaya, Horváth, and Nanni as applied to claim 1 above, and further in view of Nesteruk {S. Nesteruk et al., "XtremeAugment: Getting More From Your Data Through Combination of Image Collection and Image Augmentation," in IEEE Access, vol. 10, pp. 24010-24028, 2022, doi: 10.1109/ACCESS.2022.3154709}
Claim 4
In regards to claim 4, Anaya is not relied upon to disclose wherein the camera is configured to capture the images at regular intervals.
Nesteruk is an analogous reference from the same field of data augmentation for training neural networks and teaches a system, comprising:
a camera configured to capture two or more images of one or more objects that are stationary
{see Section IVA Data Collection, Figs.4-6 including camera(s) capturing images of stationary objects (e.g. apples)};
an artificial intelligence (AI) system operatively coupled to the camera, wherein the images of the objects form an image training set that is used by the artificial intelligence (AI) system to develop a machine learning model for the objects
{see abstract, Section 1, deep learning (DL) convolutional neural network (AI system) running a DL AI machine learning model. See also Introduction, computer vision tasks (machine learning model) including object detection and pose estimation as well as exemplary applications to digital agriculture. See also Section IVA XtremeAugment system includes a Raspberry Pi4 running an AI model for computer vision trained by an image training set via backpropagation};
wherein the image training set has one or more modified images with at least one image property different from the other images
{See Sections IIIA Hardware Dataset Augmentation (HAD) and/or IIIB Object-Based Augmentation, Section IV further describing HDA as including controlling the lighting condition such that the system takes sequential images from the cameras under two different lighting conditions. See also Fig. 5 illustrating various Object Based Augmentations such as image contrast, brightness, and saturation.
Nesteruk also discloses wherein the camera is configured to capture the images at regular intervals {see above cites and also including Fig. 4 showing images captured each day (a regular interval also corresponding to a time-lapse)}.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Anaya which already discloses creating modified images of an object using a camera mounted at a fixed position to form a static field of view with the same viewing angle in which the modified images form an image training set for machine learning model AI system and which is adapted to various robotic applications including robots using computer vision which would benefit from being trained by modified images of the image training set such that the system includes wherein the camera is configured to capture the images at regular intervals as taught by Nesteruk Horvath because Nesteruk motivates the use of data augmentation for DL models, because such models heavily rely on comprehensive data sets and that increasing the diversity of the training data improves model generalization, because there is a reasonable expectation of success, and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 7
In regards to claim 7, Anaya is not relied upon to disclose but Nesteruk teaches wherein the camera is configured to create the modified images by taking the images through a time lapse approach {see above cites and also including Fig. 4 showing images captured each day (a regular interval also corresponding to a time-lapse)}.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Anaya which already discloses creating modified images of an object using a camera mounted at a fixed position to form a static field of view with the same viewing angle in which the modified images form an image training set for machine learning model AI system and which is adapted to various robotic applications including robots using computer vision which would benefit from being trained by modified images of the image training set such that the system includes wherein the camera is configured to capture the images at regular intervals, wherein the camera is configured to create the modified images by taking the images through a time lapse approach as taught by Nesteruk because Nesteruk motivates the use of data augmentation for DL models, because such models heavily rely on comprehensive data sets and that increasing the diversity of the training data improves model generalization, because there is a reasonable expectation of success, and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 17
The rejection of system claim 7 above applies mutatis mutandis to the corresponding limitations of method claim 17.
Claim 19
In regards to claim 19, Anaya discloses wherein the modifying the images occurs during
Nesteruk teaches wherein the modifying the images occurs during and after the capturing {Sections IIIA Hardware Dataset Augmentation (HDA) which occurs during the capturing and IIIB Object-Based Augmentation which occurs after the capturing. See also Section IV further describing HDA as including controlling the lighting condition such that the system takes sequential images from the cameras under two different lighting conditions. See also Fig. 5 illustrating various Object Based Augmentations such as image contrast, brightness, and saturation}.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Anaya which already discloses creating modified images of an object using a camera mounted at a fixed position to form a static field of view with the same viewing angle in which the modified images form an image training set for machine learning model AI system and which is adapted to various robotic applications including robots using computer vision which would benefit from being trained by modified images of the image training set such that the system includes wherein the modifying the images occurs during and after the capturing as taught by Nesteruk because Nesteruk motivates the use of data augmentation for DL models, because such models heavily rely on comprehensive data sets and that increasing the diversity of the training data improves model generalization, because there is a reasonable expectation of success, and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claims 2 and 8-10 are rejected under 35 U.S.C. 103 as being unpatentable over Anaya, Horváth, and Nanni as applied to claim 1 above, and further in view of Kozak {Kozák V, Sushkov R, Kulich M, Přeučil L. Data-Driven Object Pose Estimation in a Practical Bin-Picking Application. Sensors (Basel). 2021 Sep 11;21(18):6093. doi: 10.3390/s21186093. PMID: 34577303; PMCID: PMC8473210}.
Claims 2 and 8-10
In regards to claims 2 and 8-10, Anaya is not relied upon to disclose the numerical range values of the number of images in the image training set.
Kozak is a highly analogous reference from the same field of machine learning and robotics and solves the same or similar problem of preventing overfitting of the machine learning model by using data augmentation. See below cites.
Kozak more specifically teaches, inter alia, the following elements:
an artificial intelligence (AI) system operatively coupled to the camera, wherein the images of the objects form an image training set that is used by the artificial intelligence (AI) system to develop a machine learning model for the objects
{see the vision-based object detection and localization system for bin picking of metallic parts as per abstract, sections 1.1, 2, 2.3, 3.3};
a robot configured to handle one or more items based on the machine learning model developed by the artificial intelligence (AI) system {see sections 2 and 3.2.3 robotic cell including collaborative UR10 robot configured to perform pick and place operations and other item handling based on machine learning model developed by the AI system}.
Kozak also teaches a variety of image training set sizes including 100 training images augmented eightfold to produce 800 training images and, alternatively to use approximately 2000 training images or an optimal number of 700 as per sections 3.3 and 4.1. As such, Kozak’s 700 optimized training images is within the claimed ranges of (claim 2) wherein the image training set includes at most 2,500 images, (claim 8) wherein the image training set includes at least 50 images, (claim 9) wherein the image training set includes at most 2,500 images, and (claim 10) wherein the image training set has 100 to 2,500 images and therefore anticipates all of these claimed ranges as per MPEP 2131.03 and the caselaw cited therein.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Anaya which already discloses data augmentation techniques and AI models are clearly adapted to various robotic applications such that the system employs image training sets having any of the above claimed ranges as clearly taught by Kozak because Kozak motivates using a value within each of these ranges as an optimal value, because there is a reasonable expectation of success, and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Anaya, Horváth, and Nanni as applied to claim 1 above, and further in view of Gurevich (US 20230081866 A1).
Claim 5
In regards to claim 5, Anaya is not relied upon to disclose wherein the camera is configured to capture the images at irregular intervals.
Gurevich is analogous art because it is reasonably pertinent to the problem faced by the inventor which is training a machine learning model using training image data sets. Gurevich also teaches wherein the camera is configured to capture the images at irregular intervals. See [0162] in which training image pairs may be randomly sampled/captured from the video to reduce the dataset.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Anaya which already discloses data augmentation techniques and AI models are clearly adapted to various robotic applications such that wherein the camera is configured to capture the images at irregular intervals as taught by Gurevich because Gurevich motivates doing so to reduce the dataset, because there is a reasonable expectation of success, and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Anaya, Horváth, and Nanni as applied to claim 1 above, and further in view of Kakinami (2008/0031514).
Claim 11
In regards to claim 11, Anaya is not relied upon to disclose wherein the camera is configured to calculate a camera calibration correction factor.
Kakinami is one many references demonstrating the conventional nature of camera calibration including calculating a camera calibration correction factor {see the homography computing means HM utilizing a calibration indices to compute a homography transform matrix for calibrating the camera (extrinsic and intrinsic parameters) as per Figs. 1-7, [[0055]-[0081]}
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Anaya which already discloses data augmentation techniques and AI models are clearly adapted to various robotic applications and which utilizes a camera for the augmented images such that wherein the camera is configured to calculate a camera calibration correction factor as taught by Kakinami because doing so improves the image quality thereby increasing the quality of the resulting image data training sets generated thereby and therefrom, because there is a reasonable expectation of success, and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claims 21 and 26 are rejected under 35 U.S.C. 103 as being unpatentable over Anaya, Horváth, and Nanni as applied to claim 15 above, and further in view of Morar (Morar A, Moldoveanu A, Mocanu I, Moldoveanu F, Radoi IE, Asavei V, Gradinaru A, Butean A. A Comprehensive Survey of Indoor Localization Methods Based on Computer Vision. Sensors (Basel). 2020 May 6;20(9):2641. doi: 10.3390/s20092641. PMID: 32384605; PMCID: PMC7249029}.
Claim 21
In regards to claim 21, Anaya is not relied upon to disclose augmented reality tags.
Morar establishes the highly conventional nature of wherein an augmented reality tag (ARTag) is mounted to the robot; determining a calibration factor with the machine learning system based on a location of the ARTag on the robot; replacing the camera with a replacement camera; and calibrating the replacement camera with the calibration factor {see abstract and Introduction and Section 2 including cameras attached to mobile entities such as robots and ARToolKit such is an open source library that uses ARTags to calibrate the camera while noting that camera calibration is typically necessary each time a camera is initially mounted or replaced}.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Anaya which already discloses creating modified images of an object using a camera mounted at a fixed position to form a static field of view with the same viewing angle in which the modified images form an image training set for machine learning model AI system and which is adapted to various robotic applications including robots using computer vision which would benefit from being trained by modified images of the image training set such wherein an augmented reality tag (ARTag) is mounted to the robot; determining a calibration factor with the machine learning system based on a location of the ARTag on the robot; replacing the camera with a replacement camera; and calibrating the replacement camera with the calibration factor as taught by Morar because camera calibration increases the accuracy of the image capturing process and increases the accuracy of the camera-robot position determination which improves robot functionality such as guidance/movement, because there is a reasonable expectation of success, and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 26
The rejection of system claim 21 above applies mutatis mutandis to the corresponding limitations of method claim 26.
Claim 24 and 31-33 are rejected under 35 U.S.C. 103 as being unpatentable over Anaya, Horváth, and Nanni as applied to claim 15 above, and further in view of Bainbridge (US 20230292647 A1).
Claim 24 and 31-33
Anaya is not relied upon to disclose (claims 24 and 31) wherein the human imperceptible image qualities include invisible portions of light spectrum captured by the camera that are invisible to humans, (claim 32), or (claim 33) wherein the invisible portions of the light spectrum include ultraviolet light.
Bainbridge is analogous art from the same field of generating training image data sets of machine learning. See Figs. 3, 4, 11 and cites below.
Bainbridge also teaches (claims 24 and 31) wherein the human imperceptible image qualities include invisible portions of light spectrum captured by the camera that are invisible to humans, (claim 32) wherein the invisible portions of the light spectrum include ultraviolet light, and (claim 33) wherein the invisible portions of the light spectrum include infrared light {see Fig. 11, [0137]-[0146] including hyperspectral images including infrared and UV light (a wide range is disclosed from 400-1000nm while noting that UV spans from 10nm to 400nm and thus overlaps this range while the NIR 840nm is within the infrared range}
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Anaya which already discloses creating modified images of an object using a camera mounted at a fixed position to form a static field of view with the same viewing angle in which the modified images form an image training set for machine learning model AI system and which is adapted to various robotic applications including robots using computer vision which would benefit from being trained by modified images of the image training set such that the image data includes wherein the human imperceptible image qualities include invisible portions of light spectrum captured by the camera that are invisible to humans, wherein the invisible portions of the light spectrum include ultraviolet light, or wherein the invisible portions of the light spectrum include infrared light as taught by Bainbridge because there is a reasonable expectation of success, and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 34 is rejected under 35 U.S.C. 103 as being unpatentable over Anaya, Horváth, and Nanni as applied to claim 1 above, and further in view of Lin {X. Lin, Y. Li, J. Zhu and H. Zeng, "DeflickerCycleGAN: Learning to Detect and Remove Flickers in a Single Image," in IEEE Transactions on Image Processing, vol. 32, pp. 709-720, 2023, doi: 10.1109/TIP.2022.3231748}.
Claim 34
In regards to claim 34, Anaya is not relied upon to disclose wherein the human imperceptible image qualities include light strobing.
Lin teaches that flickering artifacts (aka light strobing) are common due to artificial lighting caused by fluctuations in the power grid. In other words, AC driven light sources do not have a constant light output such that when images are captured at different parts of the cycle there is a light strobing or flickering artifact. See abstract and Section I. Lin also seeks to train a machine learning algorithm to remove such light strobing artifacts by capturing an image data training set in which the pixels have light strobing. See Sections I and III, Fig. 2.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Anaya which already discloses creating modified images of an object using a camera mounted at a fixed position to form a static field of view with the same viewing angle in which the modified images form an image training set for machine learning model AI system and which is adapted to various robotic applications including robots using computer vision which would benefit from being trained by modified images of the image training set such that wherein the human imperceptible image qualities include light strobing as taught by Lin because reducing such light flickering/strobing increases image quality, because there is a reasonable expectation of success, and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Shorten, Connor, and Taghi M. Khoshgoftaar. "A survey on image data augmentation for deep learning." Journal of big data 6.1 (2019): 1-48 discloses a variety of image data augmentation techniques including geometric, color space, kernel filters, random erasing. See pgs 7-16.
Metzler, Johannes, Fouad Bahrpeyma, and Dirk Reichelt. "An end to end workflow for synthetic data generation for robust object detection." 2023 IEEE 21st International Conference on Industrial Informatics (INDIN). IEEE, 2023 discloses a workflow for generating augmented image data sets and discloses the conventional nature of manually generating randomized conditioned images of real objects. See abstract and methodology sections.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael R Cammarata whose telephone number is (571)272-0113. The examiner can normally be reached M-Th 7am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at 571-272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL ROBERT CAMMARATA/Primary Examiner, Art Unit 2667
Prosecution Insights