DETAILED CORRESPONDENCE
This action is in response to the filing of the Application filed on 12/16/2024.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined
under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 – 7, 11, 13 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Takahashi (US 20160098032 A1).
Claim 1, Takahashi discloses a robot teaching device for a visual inspection system, the visual inspection system comprising an imager to image a workpiece, an illuminator to emit illumination light to the workpiece [see Fig 2, p0051 - an imaging unit 101, a mount base 102, an illumination source 103, and a robot 104. These components are operated under the control of a controller 200. The imaging unit 101 is an image-capturing device directed to a region containing a work WK loaded on the robot 104];
and a robot to move the imager and the illuminator, or the workpiece to inspect an appearance of the workpiece, the robot teaching device comprising: [see at least p0005, p0054 - The robot 104 changes the position and posture of the work WK, such that the imaging unit 101 can capture its images from different angles. The resulting pictures taken with various positions and postures of the work WK enable analyzing the illumination pattern in a larger area of its surface; teaching system allows the operator to execute a teaching task in a virtual space based on 3D CAD techniques, without actually operating the visual inspection device, but only by watching virtual pictures on a monitor screen. The proposed way of teaching is useful in previously selecting and checking an appropriate motion path of a camera, so that it will visit a plurality of inspection points without making the camera-carrying robot collide against the work];
a display [see display 250 – Fig 2];
Takahashi does not specifically disclose a processing unit to acquire a state of reflection of the illumination light on the workpiece by simulation and display an image relating to an acquired state of reflection of the illumination light on the display, when the robot is taught on the display to operate.
However, Takahashi does teach the simulator 304 executes an optical simulation about the light that the illumination 312 produces and calculates a picture representing what the imaging unit 311 would see in its field of view. For this picture, the simulator 304 may use ray tracing, radiosity, photon mapping, environment mapping, or other kind of rendering method. The picture is then passed to the secondary data generation unit 305. In response, the secondary data generation unit 305 calculates in what surface region of the work WKv the illumination is reflected. This region will be referred to as a “reflection region.” The secondary data generation unit 305 also calculates which part of the work surface has to be removed from the inspection view field. The secondary data generation unit 305 then produces secondary data by removing the calculated part from the reflection region.
Further teaching, the componentization unit 306 then sends secondary data indicating the calculated region to the controller 200. Suppose, for example, that the work WKv has changed its posture. In this case, the componentization unit 306 reads previous component data from the storage unit 301 and converts that component data into a region of the current picture taken in the virtual device 310. The componentization unit 306 then sends secondary data indicating the calculated region to the controller 200. The controller 200 then displays the received secondary data as an overlay on the current picture of the real device 100, (here teaching that the reflection on the illuminated object is in a state whereby the user can see optimal reflection, and inspect accordingly) so that the operator may understand the precise range that was inspected before the posture change [see p0090 – p0092].
It would have been obvious before the effective date of the claimed invention to one of ordinary skill in the art to modify the device in Takahashi to include and display an image relating to an acquired state of reflection of the illumination light on the display, when the robot is taught on the display to operate, as suggested and taught by Takahashi with a reasonable expectation of success, for the purpose of providing the imaging appearance of a flaw varies depending on a positional relationship of a camera, a workpiece, and an illumination; Accordingly, in order to reduce the number of flaws that are overlooked, imaging needs to be performed while a positional relationship of a camera, a workpiece, and an illumination is changed in each position so that the operator may understand the precise range that was inspected before the posture change.
Claim 2, Takahashi discloses the robot teaching device according to claim 1, wherein the processing unit is operable to acquire a state of reflection of the illuminator in a captured image of the workpiece captured by the imager as the state of reflection of the illumination light, and display the captured image of the workpiece including an acquired state of reflection of the illuminator on the display [see Fig 12, real device picture from capturing unit 203 and the image capturing unit 203 as can be seen from this picture, the work WK has a bright surface region in which the illumination is reflected. In addition to the work WK, the picture contains some partial images of the robot 104. Actually, the robot 104 in the picture has some portions that are as bright as the illumination-reflecting surface region of the work WK].
Claim 3, Takahashi discloses the robot teaching device according to claim 2, wherein the processing unit is operable to display, on the display, the captured image of the workpiece including the state of reflection of the illuminator when the robot is in a predetermined posture [see Figs 19 - 20, p0135 – p0137, the display shows the captured image of the workpiece (Fig 19) with a state of reflection (illuminated) and D25 is secondary data which is a picture of the WK; the region seen in the secondary data D25 would not directly fit into a new picture of the same work WK after a change in the position or posture of the work WK. The original secondary data D25 per se is not usable once the work WK is moved. The componentization unit 306 therefore transforms secondary data D25 into a piece of 3D CAD data so as to make it compatible with a change in the position or posture of the work WK (here, Takahashi discloses that the original or predetermined posture of the of the WK is saved and a captured image is complete with the state of reflection)].
Claim 4, Takahashi discloses the robot teaching device according to claim 2, wherein the processing unit is operable to display, on the display, the captured image of the workpiece including the state of reflection of the illuminator when the imager and the illuminator, or the workpiece is moved by the robot along a movement path [see p0143 – 0147 and Figs 21 – 24 showing the real display with image and different reflection of the illumination, also once a new position or posture of the work WK is optimized for inspection, the luminous energy level of the illumination source 103 is automatically adjusted using secondary data during the inspection or teaching].
Claim 5, Takahashi discloses the robot teaching device according to claim 1, wherein the processing unit is operable to acquire a state of reflection of the illuminator in a captured image of the workpiece captured by the imager as the state of reflection of the illumination light, acquire an inspection region in which the workpiece is inspectable based on an acquired state of reflection of the illuminator, and display an acquired inspection region superimposed on the workpiece displayed on the display [see Figs 21 – 24 and p0146 – p0147 teaching the produced two pieces of secondary data D23 and D24 are then sent from the secondary data generation unit 305 to the controller 200. In response, the display control unit 207 in the controller 200 displays a picture of the real device 100 on the monitor unit 250, with an overlay of a past inspection range indicated by the secondary data D23 as indicated by the broken-line frame in FIG. 24; the described overlay of an inspection range based on secondary data is applicable to both the real device pictures and virtual device pictures].
Claim 6, Takahashi discloses the robot teaching device according to claim 5, wherein the processing unit is operable to acquire a band-shaped inspection region when the imager and the illuminator, or the workpiece is moved by the robot along a movement path, and display an acquired band-shaped inspection region superimposed on the workpiece displayed on the display [see at least Fig 17 below - luminous area level adjustment showing the band-shaped inspection region (Af) superimposed on the WK displayed].
PNG
media_image1.png
633
443
media_image1.png
Greyscale
Claim 7, Takahashi discloses the robot teaching device according to claim 6, wherein the processing unit is operable to acquire a plurality of band-shaped inspection regions corresponding to a plurality of movement paths, and display an acquired plurality of band-shaped inspection regions superimposed on the workpiece displayed on the display [see at least Fig 17 below - luminous area level adjustment showing the band-shaped inspection region (Af) superimposed on the WK displayed].
Takahashi does not specifically teach “plurality of band shaped inspection regions”, however it would have been obvious before the effective date of the claimed invention to one of ordinary skill in the art to modify the device in Takahashi to include “plurality of band shaped inspection regions”, as suggested and taught by Takahashi with a reasonable expectation of success, for the purpose of providing the imaging appearance of a flaw varies depending on a positional relationship of a camera, a workpiece, and an illumination; Accordingly, in order to reduce the number of flaws that are overlooked, imaging needs to be performed while a positional relationship of a camera, a workpiece, and an illumination is changed in each position so that the operator may understand the precise range that was inspected before the posture change.
Claim 11, Takahashi discloses the robot teaching device according to claim 1, wherein the processing unit is operable to acquire the state of reflection of the illumination light by the simulation based on at least one of a color and a reflectance of a surface of the workpiece [see Figs 14 – 16, p0130 - the secondary data generation unit 305 further extracts an illuminated area representing a reflection of the illumination (S3). Specifically, the secondary data generation unit 305 extracts red pixels from the picture of the virtual device 310 when a reflection is emphasized with that color. The extracted pixels form an illuminated area. Referring to the examples of FIG. 14, illuminated area data D22 indicates an illuminated area with a color of white on the black background. This illuminated area data D22 is another example of secondary data D2].
Claim 13, Takahashi discloses a visual inspection system comprising: an imager to image a workpiece; an illuminator to emit illumination light to the workpiece; [see Fig 2, p0051 - an imaging unit 101, a mount base 102, an illumination source 103, and a robot 104. These components are operated under the control of a controller 200. The imaging unit 101 is an image-capturing device directed to a region containing a work WK loaded on the robot 104];
a robot to move the imager and the illuminator, or the workpiece to inspect an appearance of the workpiece; and a robot teaching device; wherein the robot teaching device includes: [see at least p0005, p0054 - The robot 104 changes the position and posture of the work WK, such that the imaging unit 101 can capture its images from different angles. The resulting pictures taken with various positions and postures of the work WK enable analyzing the illumination pattern in a larger area of its surface; teaching system allows the operator to execute a teaching task in a virtual space based on 3D CAD techniques, without actually operating the visual inspection device, but only by watching virtual pictures on a monitor screen. The proposed way of teaching is useful in previously selecting and checking an appropriate motion path of a camera, so that it will visit a plurality of inspection points without making the camera-carrying robot collide against the work];
a display [see display 250 – Fig 2];
Takahashi does not specifically disclose and a processing unit to acquire a state of reflection of the illumination light on the workpiece by simulation and display an image relating to an acquired state of reflection of the illumination light on the display, when the robot is taught on the display to operate.
However, Takahashi does teach the simulator 304 executes an optical simulation about the light that the illumination 312 produces and calculates a picture representing what the imaging unit 311 would see in its field of view. For this picture, the simulator 304 may use ray tracing, radiosity, photon mapping, environment mapping, or other kind of rendering method. The picture is then passed to the secondary data generation unit 305. In response, the secondary data generation unit 305 calculates in what surface region of the work WKv the illumination is reflected. This region will be referred to as a “reflection region.” The secondary data generation unit 305 also calculates which part of the work surface has to be removed from the inspection view field. The secondary data generation unit 305 then produces secondary data by removing the calculated part from the reflection region.
Further teaching, the componentization unit 306 then sends secondary data indicating the calculated region to the controller 200. Suppose, for example, that the work WKv has changed its posture. In this case, the componentization unit 306 reads previous component data from the storage unit 301 and converts that component data into a region of the current picture taken in the virtual device 310. The componentization unit 306 then sends secondary data indicating the calculated region to the controller 200. The controller 200 then displays the received secondary data as an overlay on the current picture of the real device 100, (here teaching that the reflection on the illuminated object is in a state whereby the user can see optimal reflection, and inspect accordingly) so that the operator may understand the precise range that was inspected before the posture change [see p0090 – p0092].
It would have been obvious before the effective date of the claimed invention to one of ordinary skill in the art to modify the device in Takahashi to include and a processing unit to acquire a state of reflection of the illumination light on the workpiece by simulation and display an image relating to an acquired state of reflection of the illumination light on the display, when the robot is taught on the display to operate, as suggested and taught by Takahashi with a reasonable expectation of success, for the purpose of providing the imaging appearance of a flaw varies depending on a positional relationship of a camera, a workpiece, and an illumination; Accordingly, in order to reduce the number of flaws that are overlooked, imaging needs to be performed while a positional relationship of a camera, a workpiece, and an illumination is changed in each position so that the operator may understand the precise range that was inspected before the posture change.
Claim 14, Takahashi discloses a robot teaching method for a visual inspection system, the visual inspection system comprising an imager to image a workpiece, an illuminator to emit illumination light to the workpiece [see Fig 2, p0051 - an imaging unit 101, a mount base 102, an illumination source 103, and a robot 104. These components are operated under the control of a controller 200. The imaging unit 101 is an image-capturing device directed to a region containing a work WK loaded on the robot 104];
and a robot to move the imager and the illuminator, or the workpiece to inspect an appearance of the workpiece, the robot teaching method comprising, when the robot is taught on the display to operate [see at least p0005, p0054 - The robot 104 changes the position and posture of the work WK, such that the imaging unit 101 can capture its images from different angles. The resulting pictures taken with various positions and postures of the work WK enable analyzing the illumination pattern in a larger area of its surface; teaching system allows the operator to execute a teaching task in a virtual space based on 3D CAD techniques, without actually operating the visual inspection device, but only by watching virtual pictures on a monitor screen. The proposed way of teaching is useful in previously selecting and checking an appropriate motion path of a camera, so that it will visit a plurality of inspection points without making the camera-carrying robot collide against the work];
a display [see display 250 – Fig 2];
Takahashi does not specifically teach acquiring a state of reflection of the illumination light on the workpiece by simulation; and displaying an image relating to an acquired state of reflection of the illumination light on the display.
However, Takahashi does teach the simulator 304 executes an optical simulation about the light that the illumination 312 produces and calculates a picture representing what the imaging unit 311 would see in its field of view. For this picture, the simulator 304 may use ray tracing, radiosity, photon mapping, environment mapping, or other kind of rendering method. The picture is then passed to the secondary data generation unit 305. In response, the secondary data generation unit 305 calculates in what surface region of the work WKv the illumination is reflected. This region will be referred to as a “reflection region.” The secondary data generation unit 305 also calculates which part of the work surface has to be removed from the inspection view field. The secondary data generation unit 305 then produces secondary data by removing the calculated part from the reflection region [see p0090 – p0092].
It would have been obvious before the effective date of the claimed invention to one of ordinary skill in the art to modify the device in Takahashi to include acquiring a state of reflection of the illumination light on the workpiece by simulation; and displaying an image relating to an acquired state of reflection of the illumination light on the display, as suggested and taught by Takahashi with a reasonable expectation of success, for the purpose of providing the imaging appearance of a flaw varies depending on a positional relationship of a camera, a workpiece, and an illumination; Accordingly, in order to reduce the number of flaws that are overlooked, imaging needs to be performed while a positional relationship of a camera, a workpiece, and an illumination is changed in each position so that the operator may understand the precise range that was inspected before the posture change.
Claim(s) 8 is rejected under 35 U.S.C. 103 as being unpatentable over
Takahashi (US 20160098032 A1) in view of Harrison (US 20200060007).
Claim 8, Takahashi discloses the robot teaching device according to claim 1, but does not specifically teach further comprising: a storage to store a plurality of models of the illuminator; wherein the processing unit is operable to acquire, by the simulation, the state of reflection of the illumination light according to the illuminator of a model selected from among the plurality of models of the illuminator stored in the storage.
However, Harrison discloses an integrated plan, simulate, and order lighting design approach may include representing physical features of an environment as surfaces and edges. The surface and edge data may be processed with machine learning to generate a lighting space model of the environment, which may be used to produce a floor plan, a reflected ceiling plan, and the like of the environment. By coupling the lighting space model with an augmented reality view of the environment light sources (e.g., fixtures and the like) may be added to the lighting space model by a user placing lighting source elements (e.g., icons and the like) in the augmented reality view of the environment. In embodiments, the lighting floor plan may be at least partially configured through a user interface that allows a user to select light sources from a library of lights, indicate a position and orientation of the selected light sources, and optionally define a portion of the environment for inclusion in the floor plan [see p0115, p0450].
It would have been obvious before the effective date of the claimed invention to one of ordinary skill in the art to modify the device in Takahashi to include comprising: a storage to store a plurality of models of the illuminator; wherein the processing unit is operable to acquire, by the simulation, the state of reflection of the illumination light according to the illuminator of a model selected from among the plurality of models of the illuminator stored in the storage, as suggested and taught by Harrison, with a reasonable expectation of success, for the purpose of providing a lighting installation that leverages the intelligence of lighting fixtures, including lighting fixtures that use novel control capabilities that are coordinated with other system components as described herein. References to the platform are intended to encompass, except where the context indicates otherwise, the various methods, systems, components, modules, fixtures, data structures, workflows, and other elements that are coordinated, in various embodiments, to enable the workflow.
Claim(s) 9 is rejected under 35 U.S.C. 103 as being unpatentable over
Takahashi (US 20160098032 A1) in view of Kallakuri (US 11818508).
Claim 9, Takahashi discloses the robot teaching device according to claim 1, but is silent to further comprising: a storage to store a plurality of models of the imager; wherein the processing unit is operable to acquire a captured image of the workpiece captured by the imager according to the imager of a model selected from among the plurality of models of the imager stored in the storage.
However, Kallakuri teaches systems that identify and track puts and takes of items by subjects in real space. Further disclosing, in one embodiment, the system can generate multiple camera coverage plans that meet the coverage requirements and constraints. The system can provide these to an expert to select a best camera placement plan for placing cameras in the area of real space. The camera placement data can be stored in the camera placement database 150; wherein the initial number and initial pose of a plurality of cameras and a camera model are selected from (i) a random initialized coverage plan comprising an initial number of cameras randomly distributed in the three-dimensional real space and (ii) a proto-coverage plan comprising a received input of an initial number of cameras approximately positioned in the three-dimensional real space [see at least Col 31, ll. 58 – 63, Claim 8].
It would have been obvious before the effective date of the claimed invention to one of ordinary skill in the art to modify the device in Takahashi to include a storage to store a plurality of models of the imager; wherein the processing unit is operable to acquire a captured image of the workpiece captured by the imager according to the imager of a model selected from among the plurality of models of the imager stored in the storage, as suggested and taught by Kallakuri, with a reasonable expectation of success, for the purpose of providing computer-implemented method includes providing the improved camera coverage plan to an installer to arrange cameras to track puts and takes of items by subjects in the three-dimensional real space.
Claim(s) 12 is rejected under 35 U.S.C. 103 as being unpatentable over
Takahashi (US 20160098032 A1) in view of SATOSHI (JP2019194571A).
Claim 12, Takahashi discloses the robot teaching device according to claim 1, but not specifically wherein the workpiece includes a curved surface; and the processing unit is operable to acquire a movement path for the robot substantially perpendicular to the curved surface of the workpiece and along the curved surface of the workpiece, and display an acquired movement path on the display.
However, Satoshi discloses a curved surface shape inspection apparatus 10 inspects the curved surface shape using the confocal optical sensor 50. In the case of the confocal method, the light generated in the light source 51 passes through the objective lens 55 of the head unit 53 and is reflected by the measurement object 20. Even if the optical axis 53A of the head portion 53 is tilted by several tens of degrees with respect to the normal line n (not shown) at the first measurement point P1 of the first curved surface 21, the light scattered and reflected at the first measurement point P1 is the objective lens. Pass 55. Therefore, the light reflected at the first measurement point P1 passes through the objective lens 55 without irradiating the first measurement point P1 on the first curved surface 21 strictly perpendicularly, so that the first measurement point P1. The light reflected at the second measurement point P2 passes through the objective lens 55 without irradiating light to the second measurement point P2 of the second curved surface 22 strictly perpendicularly, the second measurement point. The distance L2 to P2 can be measured. In addition, the light reflected at the third measurement point P3 passes through the objective lens 55 without irradiating light to the third measurement point P3 of the third curved surface 23 strictly perpendicularly, and thus the third measurement point P3 [see Figure 3, p0023 – p0035].
It would have been obvious before the effective date of the claimed invention to one of ordinary skill in the art to modify the device in Takahashi to include wherein the workpiece includes a curved surface; and the processing unit is operable to acquire a movement path for the robot substantially perpendicular to the curved surface of the workpiece and along the curved surface of the workpiece, and display an acquired movement path on the display, as suggested and taught by Satoshi, with a reasonable expectation of success, for the purpose of providing an easier and more reliable way to measure a curved shaped object based on the ideal angle so that the measuring the distance does not become impossible no matter how large the curved surface.
Allowable Subject Matter
Claim 10 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The examiner has pointed out particular references contained in the prior art of record in the body of this action for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. Applicant should consider the entire prior art as applicable as to the limitations of the claims. It is respectfully requested from the applicant, in preparing the response, to consider fully the entire references as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RENEE LAROSE whose telephone number is (313)446-4856. The examiner can normally be reached on Monday - Friday 8:30am - 5:00pm EST.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abby Lin can be reached on (571) 270-3976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Renee LaRose/Examiner, Art Unit 3657
/SOHANA TANJU KHAYER/ Primary Examiner, Art Unit 3657
Prosecution Insights