DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Specification
The disclosure is objected to because of the following informalities:
Specification [0005] recites, “The processor performs the display control based on association information in which a viewpoint direction when the display control is performed based on the structure information and the image capturing direction of each of the plurality of inspection images are associated with each other.” The statement can’t be interpreted regarding the scope of “association”. Also the role of “viewpoint direction” is not interpretable.
There are other paragraph for example [0033] and [0034] recite similar disclosure which can’t be interpreted.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-30 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Independent claims 1, 29 and 30 recite, “perform the display control based on association information in which a viewpoint direction in a case in which the display control is performed based on the structure information and the image capturing direction of each of the plurality of inspection images are associated with each other.”
The limitation is not clear. The scope of “association information” is not clear. It is also not clear about of the role of viewpoint direction. Specification also describes similar language. So interpretation of claim can’t be done based on specification also.
Dependent claims 2-28 are also rejected by virtue of dependency.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4 and 29-30 are rejected under 35 U.S.C. 103 as being unpatentable over Fuerst et al. (US Pat. Pub. No. 20200012116 “Fuerst”) in view of Davis et al. (US Pat. Pub. No. 20080247635 “Davis”).
Regarding claim 1 Fuerst teaches A display control device comprising a processor (“[0006]…… The display system can include a processor or controller configured to detect and track an eye position or a head position of a user relative to the display based on processing output data of the first sensor assembly. The processor or controller also can be configured to detect and track a gaze of the user based on processing output data of the second sensor assembly”) and performing display control based on a plurality of inspection images captured by radiating radiation onto an inspection target (“[0014] Additionally, an endoscopic image or other suitable image of a surgical site from the surgical robotics camera may be displayed on the display, e.g., as part of a GUI or display window on the display……… For example, a user can focus their gaze on the images or icons shown the control or side panels to trigger application interactions (e.g., to start and stop a timer application, initiate or control an x-ray viewing tool, enlarge a view, or to initiate or control other suitable applications)”) from a plurality of different image capturing directions and structure information of the inspection target (“[0044]……The GUI may further include control panels or side panels 182 including one or more images or icons 184 related to one or more applications related to the surgical robotic system 1 (e.g., a timer application, an x-ray imaging tool, etc.). The control panel(s) 182 also can include other suitable information, such as one or more medical images (e.g., pre-operative images of patient tissue), patient data (e.g., name, medical record number, date of birth, various suitable notes, etc.), tool information (e.g., a left or right tool number, a left or right tool name, a left or right tool function, etc.)”) but is silent about structure information indicating a three-dimensional structure of the inspection target
Davis teaches display control based on images image and structure information indicating a three-dimensional structure of an inspection target (“[0016] According to various aspects of the present invention, a three-dimensional (3D) model of an object of interest also referred to herein as a virtual object, is utilized for inspection and analysis. An operator interacts with the virtual object by examining, manipulating or otherwise evaluating one or more two-dimensional (2D) and/or three-dimensional (3D) views of the virtual object in a graphic environment. During the inspection, the operator may define locations on the virtual object and associate markup tags and other electronically recorded information with the defined locations. [0018] The 3D model may thus be conceptualized as a virtual 3D image (virtual object) of a corresponding real object that may reveal features or other characteristics of the real object that may be of interest, e.g., to identify degradations, defects and other conditions observable or otherwise manipulated to become detectable as described more fully herein”);
Fuerst and Davis are analogous art as both of them are related to image visualization.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Fuerst by having structure information indicating a three-dimensional structure of an inspection target as taught by Davis.
The motivation for the above is that user can understand the relationship between 3d object and inspection target.
Fuerst modified by Davis teaches wherein the processor is configured to perform the display control based on association information in which a viewpoint direction in a case in which the display control is performed based on the structure information and the image capturing direction of each of the plurality of inspection images are associated with each other (Fuerst ABSTRACT “A processor is configured to detect and track an eye position or a head position of a user relative to the display based on processing output data of the first sensor assembly, and to detect and track a gaze of the user based on processing output data of the second sensor assembly. The processor further is configured to modify or control an operation of the display system based on the detected gaze of the user. A spatial relationship of the display also can be automatically adjusted in relation to the user based on the detected eye or head position of the user to optimize the user's visualization of three-dimensional images on the display”).
Claim 29 is directed to a method and its steps are similar in scope and functions performed by the device claim 1 and therefore claim 29 is also rejected with the same rationale as specified in the rejection of claim 1.
Claim 30 is directed to “A non-transitory computer-readable storage medium” (Fuerst “[0016] Furthermore, a method for three-dimensional visualization during robotic surgery can be provided. The method can be performed by a digital programmed processor executing instructions stored in a computer readable memory”) and its elements are similar in scope and functions performed by the device claim 1 and therefore claim 30 is also rejected with the same rationale as specified in the rejection of claim 1.
Regarding claim 2 Fuerst modified by Davis teaches wherein: the processor is configured to perform control for displaying at least one of the inspection images and a virtual inspection target based on the association information, the virtual inspection target being based on the structure information (Davis “[0050] The method 180 of using a GUI as described with reference to FIG. 4 may be implemented for example, by interacting with the interface 200 of FIG. 5 and correspondingly, the GUI 150 of FIG. 3. For example, with reference to FIGS. 3, 4 and 5 generally, a 3D model of a modeled (virtual) object is acquired. For example, an operator interacting with the interface 200 may select appropriate controls from the mode window 206, such as a "Raw Data Acquisition" control to load an image set or "Database Browse" control to perform database browsing and/or scan data filtering, e.g., to search, filter, locate and select an image set to load into the inspection system”).
Regarding claim 3 Fuerst modified by Davis teaches wherein: the processor is configured to change a display mode of the at least one of the plurality of inspection images based on the association information (Fuerst “[0014]…… The detected and tracked gaze of the user further can be used to initiate or control the applications in the control/side panels. For example, a user can focus their gaze on the images or icons shown the control or side panels to trigger application interactions (e.g., to start and stop a timer application, initiate or control an x-ray viewing tool, enlarge a view, or to initiate or control other suitable applications)”).
Regarding claim 4 Fuerst modified by Davis teaches wherein: the processor is configured to change the display mode of the inspection image based on the viewpoint direction. (Fuerst ABSTRACT “A processor is configured to detect and track an eye position or a head position of a user relative to the display based on processing output data of the first sensor assembly, and to detect and track a gaze of the user based on processing output data of the second sensor assembly. The processor further is configured to modify or control an operation of the display system based on the detected gaze of the user. A spatial relationship of the display also can be automatically adjusted in relation to the user based on the detected eye or head position of the user to optimize the user's visualization of three-dimensional images on the display”).
Claim(s) 5, 7 and 9-10 are rejected under 35 U.S.C. 103 as being unpatentable over Fuerst modified by Davis as applied to claim 4 above, and further in view of CHA et al. (US Pat. Pub. No. 20220295035 “Cha”).
Regarding claim 5 even though Fuerst modified by Davis teaches wherein: the processor is configured to: select one inspection image from the plurality of inspection images (Davis “For example, if the selected view is a 2D view representing an image file, the inspection system, e.g., via the processing module 164, maps the 2D image points, including the control points 220 identifying the location 218, to corresponding 3D global coordinates, such as using an associated range map 126 as described in greater detail herein”) but is silent about performing so based on an angle formed between the viewpoint direction and the image capturing direction, and change the display mode of the selected inspection image.
Cha teaches select an inspection image based on an angle formed between viewpoint direction and image capturing direction, and change display mode of the selected inspection image (ABSTRACT “A real-time broadcasting device includes a plurality of image capturing units arranged to capture images at the same time in different directions from the same point; an image selection unit for comparing the angle of view covered by each of the plurality of image capturing units and the gaze direction of a user terminal, and selecting an image of the image capturing unit, among the plurality of image capturing units, capturing images corresponding to the gaze direction of the user terminal; an image transmission unit for transmitting virtual reality images captured by the image capturing unit selected by the image selection unit to the user terminal in real time; and a gaze direction reception unit for receiving feedback about which direction the gaze direction of the user terminal viewing the virtual reality images is, and transmitting same to the image selection unit”);
Cha and Fuerst modified by Davis are analogous art as both of them are related to image visualization.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Fuerst modified by Davis by selecting an inspection image based on an angle formed between viewpoint direction and image capturing direction, and change display mode of the selected inspection image as taught by Cha.
The motivation for the above is to provide control to the user in selecting a specific image.
Regarding claim 7 Fuerst modified by Davis and Cha teaches wherein: the processor is configured to perform control for highlighting the selected inspection image (Davis [0054] Moreover, according to various aspects of the present invention, the operator may be able to perform operations (not shown) from the control window 206 such as displaying the main properties of the image set or viewing thumbnails of selected (highlighted/filtered) image sets”).
Regarding claim 9 Fuerst modified by Davis and Cha teaches the processor is configured to perform control for displaying only the selected inspection image of the plurality of inspection images (Davis Fig. 5 shows one selected image in the pane 214).
Regarding claim 10 Fuerst modified by Davis teaches wherein: in a case in which the processor performs control for displaying the plurality of inspection images (Fuerst Fig. 4) but is silent about the processor is configured to dispose each inspection image at a position according to an angle formed between the viewpoint direction and the image capturing direction of the inspection image.
Cha teaches dispose each inspection image at a position according to an angle formed between viewpoint direction and image capturing direction of the inspection image (ABSTRACT “A real-time broadcasting device includes a plurality of image capturing units arranged to capture images at the same time in different directions from the same point; an image selection unit for comparing the angle of view covered by each of the plurality of image capturing units and the gaze direction of a user terminal, and selecting an image of the image capturing unit, among the plurality of image capturing units, capturing images corresponding to the gaze direction of the user terminal; an image transmission unit for transmitting virtual reality images captured by the image capturing unit selected by the image selection unit to the user terminal in real time; and a gaze direction reception unit for receiving feedback about which direction the gaze direction of the user terminal viewing the virtual reality images is, and transmitting same to the image selection unit”);
Cha and Fuerst modified by Davis are analogous art as both of them are related to image visualization.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Fuerst modified by Davis by disposing each inspection image at a position according to an angle formed between viewpoint direction and image capturing direction of the inspection image as taught by Cha.
The motivation for the above is organize the inspection images in an order on a display for better understanding of the images by the user.
Claim(s) 6 is rejected under 35 U.S.C. 103 as being unpatentable over Fuerst modified by Davis as applied to claim 4 above, and further in view of Hall et al. (US Pat. Pub. No. 20030181809 “Hall”).
Regarding claim 6 Fuerst modified by Davis is silent about select one inspection image from the plurality of inspection images based on a similarity between a two-dimensional image and the inspection image; and changes the display mode of the selected inspection image, the two-dimensional image being obtained by projecting the structure information in the viewpoint direction.
Hall teaches select one inspection image from the plurality of inspection images based on a similarity between a two-dimensional image and the inspection image; and changes the display mode of the selected inspection image, the two-dimensional image being obtained by projecting the structure information in a direction (“[0016] Another possibility of registering the images provides for the use of two 2D x-ray images which are positioned at a certain angle, preferably 90.degree., relative to each other and in which several identical markings are identified. [0017] Another possibility is the so-called "image-based" registration. In this case, the 3D reconstructed image is used to generate a 2D projection image in the form of a digitally reconstructed radiogram (DRR) which is compared to the 2D X-ray image for similarities, and for the purpose of optimizing the registration, the similarity between the 2D projection image and the 2D X-ray image is moved by means of translation and/or rotation until the similarities reach a predetermined minimum level of similarity”);
Hall and Fuerst modified by Davis are analogous art as both of them are related to image visualization.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Fuerst modified by Davis by selecting one inspection image from the plurality of inspection images based on a similarity between a two-dimensional image and the inspection image; and changes the display mode of the selected inspection image, the two-dimensional image being obtained by projecting the structure information in the viewpoint direction similar to selecting one inspection image from plurality of inspection images based on a similarity between a two-dimensional image and the inspection image; and changes the display mode of the selected inspection image, the two-dimensional image being obtained by projecting the structure information in a direction as taught by Hall.
The motivation for the above is to provide control to user in choosing the inspection image.
Claim(s) 8 is rejected under 35 U.S.C. 103 as being unpatentable over Fuerst modified by Davis and Cha as applied to claim 5 above, and further in view of IMAMURA et al. (US Pat. Pub. No. 20190046130 “Imamura”).
Regarding claim 8 Fuerst modified by Davis and Cha is silent about change a display position of the selected inspection image.
Imamura teaches change a display position of the selected inspection image (“[0031]…… a composite image generation step of combining the camera image and a positioning index image indicating a set position of the subject, which has been set in advance with respect to the in-image cassette position, to generate a composite image and, in a case in which the in-image cassette position in the camera image is changed with movement of the electronic cassette, changing a display position of the positioning index image in the composite image with the change in the in-image cassette position; and a display control step of performing control such that the composite image is displayed on a display unit”);
Imamura and Fuerst modified by Davis are analogous art as both of them are related to image visualization.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Fuerst modified by Davis by changing a display position of the selected inspection image as taught by Imamura.
The motivation for the above is to organize content in the display to get proper focus at the inspection image by the user.
Claim(s)11 is rejected under 35 U.S.C. 103 as being unpatentable over Fuerst modified by Davis as applied to claim 4 above, and further in view of Nishii et al. (US Pat. Pub. No. 20090103676 “Nishii”).
Regarding claim 11 Fuerst modified by Davis is silent about performs control for displaying the plurality of inspection images, the processor is configured to dispose the inspection images in an order according to attribute information of each inspection image.
Nishii teaches dispose inspection images in an order according to attribute information of each inspection image (ABSTRACT “In a display control apparatus, an image receiving portion receives a plurality of images sequentially transmitted from an X-ray imaging apparatus based on a transmission order determined by a transmission-order determining portion, and a capture-order information receiving portion also receives, from the X-ray imaging apparatus, capture-order information indicating a capture order in which each of the images was captured by the X-ray imaging apparatus. The images received by the image receiving portion are stored in an image storage memory. A display controller controls displaying of the images stored in the image storage memory on a display apparatus”).
Nishii and Fuerst modified by Davis are analogous art as both of them are related to image visualization.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Fuerst modified by Davis by disposing inspection images in an order according to attribute information of each inspection image as taught by Nishii.
The motivation for the above is to organize the inspection in a the display to help user interpret the images.
Claim(s) 12 is rejected under 35 U.S.C. 103 as being unpatentable over Fuerst modified by Davis as applied to claim 4 above, and further in view of Lee (US Pat. Pub. No. 20180181196 “Lee”).
Regarding claim 12 Fuerst modified by Davis is silent about wherein: in a case in which the processor performs control for displaying the plurality of inspection images, the processor is configured to dispose the inspection images in an order according to an angle formed between the viewpoint direction and the image capturing direction of each inspection image or in an order according to a similarity between a two-dimensional image and each inspection image, the two-dimensional image being obtained by projecting the structure information in the viewpoint direction.
Lee teaches dispose the inspection images in an order according to an angle formed between the viewpoint direction and the image capturing direction of each inspection image (“[0166] When a request to play a 360-degree image is made in operation 920, the electronic device (e.g., the processor 120) may display a first partial image, which corresponds to a first direction, in the 360-degree image in operation 925. The first direction may be a reference direction in which a main camera taking the 360-degree image is pointed straight ahead. [0174] In operation 940, the electronic device may display a second partial image corresponding to the second direction. The second direction may be a direction different from the first direction. When the first partial image corresponds to a first portion of each frame of the taken image, the second partial image may correspond to a second portion of the frame. [0187] However, when the electronic device plays an image in a main traveling direction (i.e., a moving direction), in which the photographer travels, the electronic device may display a partial image according to an angle resulting from the change attributable both to the angle of the moving direction relative to the angle of the main camera direction 1110 and to the angle (Φ.sub.2, θ.sub.2) of the HMD gaze direction”);
Lee and Fuerst modified by Davis are analogous art as both of them are related to image visualization.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Fuerst modified by Davis by disposing the inspection images in an order according to an angle formed between the viewpoint direction and the image capturing direction of each inspection image as taught by Lee.
The motivation for the above is to organize the inspection in a the display to help user interpret the images.
Claim(s) 13 is rejected under 35 U.S.C. 103 as being unpatentable over Fuerst modified by Davis as applied to claim 2 above, and further in view of Mizuno (US Pat. Pub. No. 20190182470 “Mizuno”).
Regarding claim 13 Fuerst modified by Davis is silent about change a display mode of the virtual inspection target based on the image capturing direction according to a position selected within a display region of the plurality of inspection images.
Mizuno teaches change a display mode of the virtual inspection target based on the image capturing direction according to a position selected within a display region of the plurality of inspection images (“[0007] In the present invention, there is provided an information processing apparatus for causing a displaying unit to display an image related to a virtual viewpoint image generated based on a plurality of captured images obtained by capturing a capturing area with a plurality of cameras and on position and direction of a virtual viewpoint, the information processing apparatus comprising: a determining unit configured to determine, among the captured images obtained by the plurality of cameras, a display target image being an image to be displayed, based on virtual viewpoint information representing the position and direction of the virtual viewpoint; and a display controlling unit configured to cause the displaying unit to display the display target image determined by the determining unit”);
Mizuno and Fuerst modified by Davis are analogous art as both of them are related to image visualization.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Fuerst modified by Davis by changing a display mode of the virtual inspection target based on the image capturing direction according to a position selected within a display region of the plurality of inspection images as taught by Mizuno.
The motivation for the above is show better correlation between virtual image and inspection image.
Claim(s) 14 is rejected under 35 U.S.C. 103 as being unpatentable over Fuerst modified by Davis and Mizuno as applied to claim 13 above, and further in view of Lee (US Pat. Pub. No. 20180181196 “Lee”).
Regarding claim 14, Fuerst modified by Davis and Mizuno doesn’t expressly teach, wherein the processor is configured to change the viewpoint direction to a direction identical to the image capturing direction of a selected one of the inspection images.
However Lee teaches, change the viewpoint direction to a direction identical to the image capturing direction of a selected one of the inspection images (“[0187]….. As described above, when a user selects a viewpoint change to one viewpoint based on the traveling direction while an image is played, an image in a direction viewed with the HMD device based on the traveling direction may be an image corresponding to an angle resulting from the following calculation the angle (Φ.sub.1, θ.sub.1) of the main camera direction 1110”);
Lee and Fuerst modified by Davis and Mizuno are analogous art as both of them are related to image visualization.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Fuerst modified by Davis and Mizuno by changing the viewpoint direction to a direction identical to the image capturing direction of a selected one of the inspection images as taught by Lee.
The motivation for the modification is that user would be able to see corresponding side of virtual object matching with inspection image.
Claim(s) 15 is rejected under 35 U.S.C. 103 as being unpatentable over Fuerst modified by Davis and Mizuno as applied to claim 13 above, and further in view of FUKUSHI (US Pat. Pub. No. 20160132991 “Fukushi”).
Regarding claim 15, Fuerst modified by Davis and Mizuno doesn’t expressly teach, wherein: in accordance with an operation for moving a selected one of the inspection images, the processor is configured to change the viewpoint direction to a direction identical or opposite to the operation for moving the selected inspection image.
Fukushi teaches change viewpoint direction to a direction identical or opposite to the operation for moving the selected inspection image (“[0041] FIG. 4 (c) indicates a case in which the accepting unit 15 has accepted an instruction to change to the viewpoint E from a user. Specifically, the accepting unit 15 accepts input indicating the instruction to change the display of the omnidirectional image included in the display image to the viewpoint E in response to operation by a user. The operation by the user includes operation of designating the viewpoint E on the omnidirectional image 30, operation of choosing a thumbnail 401 (thumbnail of a partial omnidirectional image that is an image of a region including the viewpoint E) of the partial omnidirectional image on the viewpoint listing image 40, operation of bringing the thumbnail 41 of the partial omnidirectional image to the center of the viewpoint listing image (operation of shifting from a state of the viewpoint listing image 40 to a state of a viewpoint listing image 41) by scrolling on the viewpoint listing image 40, or the like”);
Fukushi and Fuerst modified by Davis and Mizuno are analogous art as both of them are related to image visualization.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Fuerst modified by Davis and Mizuno by changing viewpoint direction to a direction identical or opposite to the operation for moving the selected inspection image as taught by Fukushi.
The motivation for the modification is that user would be able to see virtual object with a differing view of the inspection image.
Claim(s) 17-19 and 21-23 are rejected under 35 U.S.C. 103 as being unpatentable over Fuerst modified by Davis as applied to claim 2 above, and further in view of Weiss et al. (US Pat. Pub. No. 20170206428 “Weiss”).
Regarding claim 17 Fuerst modified by Davis is silent about perform control for displaying information indicating a correspondence relationship between the structure information and the plurality of inspection images in combination with at least either the virtual inspection target or the inspection images based on the association information.
Weiss teaches perform control for displaying information indicating a correspondence relationship between the structure information and the plurality of inspection images in combination with at least either the virtual inspection target or the inspection images based on the association information (Fig. 6 “[0132]….. the digital photographic image, as shown in FIG. 6. The system can then calculate the length, width, radius, etc. of the feature shown in the digital photographic image by interpolating between X and Y grid curves in the virtual measurement space overlaid on the digital photographic image”);
Weiss and Fuerst modified by Davis are analogous art as both of them are related to image visualization.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Fuerst modified by Davis by performing control for displaying information indicating a correspondence relationship between the structure information and the plurality of inspection images in combination with at least either the virtual inspection target or the inspection images based on the association information as taught by Weiss.
The motivation for the above is to synchronization the virtual object and the inspection image in the display.
Regarding claim 18 Fuerst modified by Davis and Weiss teaches perform control for displaying an object indicating that any of the inspection images is present in a vicinity of the virtual inspection target and in a direction corresponding to the image capturing direction of the inspection image (Davis “[0062] Global coordinate points of the 3D model are then designated that characterize the location of interest e.g., a 3D subset of the larger modeled object. As will be described in greater detail below, a markup tag containing information can then be created that is associated with the location of interest. For example, if the selected view is a 2D view representing an image file, the inspection system, e.g., via the processing module 164, maps the 2D image points, including the control points 220 identifying the location 218, to corresponding 3D global coordinates, such as using an associated range map 126 as described in greater detail herein”).
Regarding claim 19 Fuerst modified by Davis and Weiss teaches perform control for highlighting the object indicating that a selected one of the inspection images is present (Davis “[0054] Moreover, according to various aspects of the present invention, the operator may be able to perform operations (not shown) from the control window 206 such as displaying the main properties of the image set or viewing thumbnails of selected (highlighted/filtered) image sets. The operator may also be able to switch to other modes such as a visualization mode using the "Visualization" control and/or an inspection mode using the "Markup/Inspection" control, which will be described in greater detail below”).
Regarding claim 21, Fuerst modified by Davis and Weiss teaches the processor is configured to perform control for displaying a feature point in combination with a selected one of the inspection images and the virtual inspection target, the feature point corresponding to where the inspection image and the virtual inspection target correspond with each other. (Davis “[0062] Global coordinate points of the 3D model are then designated that characterize the location of interest e.g., a 3D subset of the larger modeled object. As will be described in greater detail below, a markup tag containing information can then be created that is associated with the location of interest. For example, if the selected view is a 2D view representing an image file, the inspection system, e.g., via the processing module 164, maps the 2D image points, including the control points 220 identifying the location 218, to corresponding 3D global coordinates, such as using an associated range map 126 as described in greater detail herein”).
Regarding claim 22 Fuerst modified by Davis and Weiss teaches wherein: the processor is configured to perform control for displaying an object indicating a discontinuity in combination with a location thereof on the virtual inspection target corresponding to the location of the discontinuity on a selected one of the inspection images. (Weiss “[0030] In this variation, when a defect on a unit is detected in an earliest image of the assembly unit, the system can flag a unit serial number corresponding to the image in which the defect was detected and then insert a defect flag into the virtual representation of the assembly at a particular optical inspection station at which this image was captured. The system can thus visually indicate to a user through the virtual representation of the assembly line that the defect on the assembly unit occurred between the particular optical inspection station and a second optical inspection station immediately preceding the particular optical inspection station in the assembly line. Furthermore, if the system detects defects shown in multiple images captured at a particular optical inspection station, the system can identify defects of the same type (e.g., similar scratches in the same area on a housing across multiple units) and incorporate a counter for defects of the same defect type into the virtual representation of the assembly line. The system can also visually represent frequency, type, and/or position of detected defects across a batch of units passing through one or more optical inspection stations, such as in the form of a heatmap For example, the system can generate or access a virtual representation of a “nominal” e.g., “generic”) unit, calculate a heatmap containing a visual representation of aggregate defects detected in like units passing through a single optical inspection station or passing through multiple optical inspection stations in the assembly line, and then present the heatmap overlayed on the virtual representation of the nominal unit within the user interface”);
Regarding claim 23 Fuerst modified by Davis is silent about wherein: positional information of a discontinuity appearing in any of the inspection images and positional information of the discontinuity in the structure information are associated with each other, and the processor is configured to change at least one of a display mode of the virtual inspection target or a display mode of the inspection image based on the positional information of the discontinuity selected in the inspection image or the structure information.
Weiss teaches positional information of a discontinuity appearing in any of the inspection images and positional information of the discontinuity in the structure information are associated with each other, and the processor is configured to change at least one of a display mode of the virtual inspection target or a display mode of the inspection image based on the positional information of the discontinuity selected in the inspection image or the structure information (“[0030] In this variation, when a defect on a unit is detected in an earliest image of the assembly unit, the system can flag a unit serial number corresponding to the image in which the defect was detected and then insert a defect flag into the virtual representation of the assembly at a particular optical inspection station at which this image was captured. The system can thus visually indicate to a user through the virtual representation of the assembly line that the defect on the assembly unit occurred between the particular optical inspection station and a second optical inspection station immediately preceding the particular optical inspection station in the assembly line. Furthermore, if the system detects defects shown in multiple images captured at a particular optical inspection station, the system can identify defects of the same type (e.g., similar scratches in the same area on a housing across multiple units) and incorporate a counter for defects of the same defect type into the virtual representation of the assembly line. The system can also visually represent frequency, type, and/or position of detected defects across a batch of units passing through one or more optical inspection stations, such as in the form of a heatmap For example, the system can generate or access a virtual representation of a “nominal” e.g., “generic”) unit, calculate a heatmap containing a visual representation of aggregate defects detected in like units passing through a single optical inspection station or passing through multiple optical inspection stations in the assembly line, and then present the heatmap overlayed on the virtual representation of the nominal unit within the user interface”);
Weiss and Fuerst modified by Davis are analogous art as both of them are related to image visualization.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Fuerst modified by Davis by having positional information of a discontinuity appearing in any of the inspection images and positional information of the discontinuity in the structure information are associated with each other, and the processor is configured to change at least one of a display mode of the virtual inspection target or a display mode of the inspection image based on the positional information of the discontinuity selected in the inspection image or the structure information as taught by Weiss.
The motivation for the above is to provide the user to view matching/related inspection image and view of inspection target.
Claim(s) 24 is rejected under 35 U.S.C. 103 as being unpatentable over Fuerst modified by Davis and Weiss as applied to claim 23 above, and further in view of Haik et al. (US Pat. Pub. No. 20190335041 “Haik”).
Regarding claim 24 Fuerst modified by Davis and Weiss is silent about change the display mode of the inspection image based on a distance between a location of the discontinuity selected from a plurality of discontinuities and a location of a discontinuity other than the selected discontinuity.
Haik teaches change the display mode of the inspection image based on a distance between a location of the discontinuity selected from a plurality of discontinuities and a location of a discontinuity other than the selected discontinuity (“[0045] The processing circuitry 404 comprises an image analyser 406, which, in operation, locates defects in a plurality of printed images and a defect assessment module 408, which, in operation, identifies defects having a spatial periodicity (i.e. ‘periodic defects’, or defects having a repeated or recurring separation distance) by measuring a separation of defects on multiple scanned images. For example, this may comprise considering the spacing between a pluralities of ‘pairs’ of defect”);
Haik and Fuerst modified by Davis and Weiss are analogous art as both of them are related to image processing.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Fuerst modified by Davis and Weiss by changing the display mode of the inspection image based on a distance between a location of the discontinuity selected from a plurality of discontinuities and a location of a discontinuity other than the selected discontinuity as taught by Haik.
The motivation for the above is that user can differentiate between multiple discontinuities of the object.
Claim(s) 25 is rejected under 35 U.S.C. 103 as being unpatentable over Fuerst modified by Davis and Weiss as applied to claim 23 above, and further in view of Chen et al. (US Pat. Pub. No. 20130082174 “Chen”).
Regarding claim 25 Fuerst modified by Davis and Weiss is silent about change the display mode of the inspection image based on a ratio between an average value of pixel values in a region of the selected discontinuity and an average value of pixel values in a region surrounding the region of the discontinuity.
Chen teaches change the display mode of the inspection image based on a ratio between an average value of pixel values in a region of the selected discontinuity and an average value of pixel values in a region surrounding the region of the discontinuity (“[0030] The background area surrounding the defect may be selected in an automated or manual manner. For example, the background area may be selected as those pixels within a certain range from the defect area, but not within the defect area. The height difference 114 between the defect area and the background area surrounding the defect area may be determined by calculating an average height of the defect area, calculating the average height of the background area, and calculating a difference between the two average heights. The measure of the height variation 116 within the defect area and the measure of the height variation 118 within the background area may be statistical measures. An example of a statistical measure of height variation is the standard deviation of the height”);
Chen and Fuerst modified by Davis and Weiss are analogous art as both of them are related to image processing.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Fuerst modified by Davis and Weiss by changing the display mode of the inspection image based on a ratio between an average value of pixel values in a region of the selected discontinuity and an average value of pixel values in a region surrounding the region of the discontinuity as taught by Chen.
The motivation for the above is to show difference between discontinuity and surrounding region to help user’s visualization of defective elements.
Claim(s) 26 is rejected under 35 U.S.C. 103 as being unpatentable over Fuerst modified by Davis and Weiss as applied to claim 23 above, and further in view of Smolic et al. (US Pat. Pub. No. 20110261050 “Smolic”).
Regarding claim 26 Fuerst modified by Davis and Weiss is silent about change the display mode of the inspection image based on a degree of smoothness of pixel values in a region surrounding the selected discontinuity.
Smolic teaches change the display mode of the inspection image based on a degree of smoothness of pixel values in a region surrounding the selected discontinuity.
(“[0107] This means that original views V1 and V5 span 20 cm, a distance that is difficult to handle with DIBR. Severe artifacts are known to occur especially along object borders with large depth discontinuities. On the other hand areas with smooth depth variations can be projected very reliably to virtual intermediate views. This implies separate processing of depth discontinuities and smooth depth regions. Depth discontinuities can be found easily within the depth images using edge detection algorithms.”);
Smolic and Fuerst modified by Davis and Weiss are analogous art as both of them are related to image processing.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Fuerst modified by Davis and Weiss by changing the display mode of the inspection image based on a degree of smoothness of pixel values in a region surrounding the selected discontinuity as taught by Smolic.
The motivation for the above is to show difference between discontinuity and surrounding region to properly recognize discontinuity.
Claim(s) 27 is rejected under 35 U.S.C. 103 as being unpatentable over Fuerst modified by Davis and Weiss as applied to claim 23 above, and further in view of DEMIZU et al. (US Pat. Pub. No. 20180276330 “Demizu”).
Regarding claim 27 Fuerst modified by Davis is silent about the processor is configured to change the viewpoint direction such that the selected discontinuity is located at a center and at a front side of a screen.
Demizu teaches change the viewpoint direction such that the selected discontinuity is located at a center and at a front side of a screen (“[0156] In this way, a viewpoint with a high priority or a viewpoint with a high similarity of the image feature amount is adopted as a new viewpoint for the portion of interest. In the three-dimensional assembly model whose design has progressed, the visibility state focused on the defective portion at the time of occurrence of a defect is reproduced and displayed on the output unit (the display unit) 40 with the viewpoint changed to the adopted new viewpoint (operation S7 in FIG. 12)”);
Demizu and Fuerst modified by Davis and Weiss are analogous art as both of them are related to image processing.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Fuerst modified by Davis and Weiss by changing viewpoint direction such that the selected discontinuity is located at a center and at a front side of a screen as taught by Demizu.
The motivation for the above is to provide better focus to discontinuity element to the user.
Claim(s) 28 is rejected under 35 U.S.C. 103 as being unpatentable over Fuerst modified by Davis and Weiss as applied to claim 23 above, and further in view of Goel et al. (US Pat. Pub. No. 20180225867 “Goel”).
Regarding claim 28 Fuerst modified by Davis is silent about change the viewpoint direction such that the inspection target has a smallest thickness in the viewpoint direction at a location of the selected discontinuity.
Goel teaches change the viewpoint direction such that the inspection target has a smallest thickness in the viewpoint direction at a location of the selected discontinuity (Fig. 2 shows defect with smallest thickness “[0160]…….The server(s) 806 can also provide one or more digital images of the at least one exterior viewpoint and/or the at least one interior viewpoint for display to the client device 802a (e.g., via the digital image engine 710). In particular, the server(s) 806 can provide one or more digital images that illustrate the three-dimensional model and defects identified in the three-dimensional model. Moreover, the server(s) can assign colors (e.g., via the color manager 708) to vertices corresponding to each of the defects, such that the digital images display, at the client device 802a, adjacent connected vertices in different colors.”);
Goel and Fuerst modified by Davis and Weiss are analogous art as both of them are related to image processing.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Fuerst modified by Davis and Weiss by changing viewpoint direction such that the inspection target has a smallest thickness in the viewpoint direction at a location of the selected discontinuity as taught by Goel.
The motivation for the above is to better visualization of the defect in an image.
Allowable Subject Matter
Claims 16 and 20 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
Regarding claim 16, the combination of prior arts fail to expressly teach, change the viewpoint direction such that a relative positional relationship between the selected position and the inspection images disposed in a vicinity of the selected position becomes closer to a relative positional relationship between the viewpoint direction and the inspection images disposed in the vicinity.
Regarding claim 20,The combination of prior arts fail to expressly teach, perform control for displaying a selected one of the inspection images in combination with a lattice-like auxiliary line and performs control for displaying the virtual inspection target in combination with a lattice-like auxiliary line having a same pitch as the auxiliary line in a plane orthogonal to a direction corresponding to the image capturing direction of the selected inspection image.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SAPTARSHI MAZUMDER whose telephone number is (571)270-3454. The examiner can normally be reached 8 am-4 pm PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said Broome can be reached at (571)272-2931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SAPTARSHI MAZUMDER/ Primary Examiner, Art Unit 2612
Prosecution Insights