Skip to content

mkazhdan/TextureSignalProcessing

BranchesTags

Repository files navigation

Gradient Domain Texture Processing (Version 9.50)

links executables usage library compilation changes support
This software supports gradient-domain signal processing within a texture atlas. Supported applications include:
  • (localized) texture smoothing and sharpening
  • vector-field visualization akin to line-integral convolution
  • computation of single-source geodesics
  • simulation of reaction-diffusion following the Gray-Scott model
  • dilation of texture into the gutter
  • masking of gutter/interior/boundary texels
  • solving for the smoothest interpolant within a prescribed subset of texels
LINKS
EXECUTABLES
    These applications support reading in textures meshes in one of two formats:
    • PLY (to support multiple charts, texture is assumed to be encoded at wedges/corners rather than vertices.)
    • Wavefront OBJ
    Input textures are assumed to be images in png, jpg, or jpeg format.
    TextureFiltering: Supports the (localized) smoothing and sharpening of a texture by solving a screened Poisson equation which gives the signal whose values match the input and whose gradients match the modulated gradients of the input. If no output texture is specified, the executable will launch an interactive viewer that supports local "painting" of gradient modulation values and prescription of a global interpolation weight.
    In the interactive viewer the modulation can be set globally by dragging the slider on the top left.
    The modulation can be set locally by holding the [SHIFT] key down and either dragging with the left mouse button (to smooth) or the right mouse button (to sharpen).
    --in <input mesh and texture names>
    These two strings specify the the names of the mesh and the texture image.
    [--out <output texture>]
    This string is the name of the file to which the processed texture will be written.
    [--outVCycles <output v-cycles>]
    This integer specifies the number of v-cycles to use if the processed texture is output to a file and a direct solver is not used. The default value for this parameter is 6.
    [--interpolation <interpolation weight>]
    This floating point values gives the interpolation weight.
    The default value for this parameter is 1000.
    [--modulation <gradient modulation>]
    This floating point values gives the (uniform) gradient modulation.
    The default value for this parameter is 1.
    [--jitter <random seed>]
    If specified, this integer value is used to seed the random number generation for jittering. (This is used to avoid singular situations when mesh vertices fall directly on edges in the texture grid. In such a situation, the executable will issue a warning "Zero row at index ...".)
    [--useDirectSolver]
    If enabled, this flag specifies that a direct solver should be used (instead of the default multigrid solver).
    TextureStitching: Supports the stitching together of multiple (possibly overlapping) textures by solving a screened Poisson equation with value constraints defined by the input texture. The interactive viewer runs in two modes:
    1. A user specifies a single composite texture and (optionally) a mask file indicating when texels in the composite come from the same source. In this case gradient constraints are obtained by copying gradients from the composite whenever the two texels defining an edge come from the same source, and setting the gradient constraint to zero along edges coming from different sources. If no mask file is provided, a default mask is created by assigning texels the same color if and only if they are covered by the same chart.
      The viewer shows the stitched texture on the left and the composite texture on the right.
    2. A user specifies multiple partial texture files and corresponding confidence masks. In this case gradient constraints are obtained by blending gradients from the different inputs, weighted by confidence, and setting gradients to zero in regions where there are no textures with non-zero confidence. The viewer shows the stitched texture on the left and a partial texture on the right. The user can selectively replace blended value/gradient constraints with the values/gradients from the partial texture by holding the [SHIFT] key down and dragging over the region to be in-painted.
    --in <input mesh and composite texture>
    These two strings specify the names of the mesh and the texture image.
    [--mask <input mask>]
    This string specifies the name of the mask image.
    Black pixels in the mask file should be used to denote regions where the texel value is unkown. (Results may be unpredictable if it is encoded using lossy compression.)
    [--out <output texture>]
    This string is the name of the file to which the stitched texture will be written.
    [--outVCycles <output v-cycles>]
    This integer specifies the number of v-cycles to use if the stitched texture is output to a file and a direct solver is not used.
    The default value for this parameter is 6.
    [--interpolation <interpolation weight>]
    This floating point values gives the interpolation weight.
    The default value for this parameter is 100.
    [--jitter <random seed>]
    If specified, this integer value is used to seed the random number generation for jittering. (This is used to avoid singular situations when mesh vertices fall directly on edges in the texture grid. In such a situation, the executable will issue a warning "Zero row at index ...".)
    [--useDirectSolver]
    If enabled, this flag specifies that a direct solver should be used (instead of the default multigrid solver).
    [--multi]
    If enabled, this flag specifies that the second and third arguments to the --in parameter are to be interpreted as format specifiers for the textures confidence map files.
    Note: If this flat is enabled, the input masks must be specified using the --mask parameter.
    LineIntegralConvolution: Supports the line-integral-convolution visualization of a vector-field through anisotropic diffusion by defining a new metric on the surface that stretches distances along the vector-field values, diffusing a random color texture with respect to the new anisotropic metric, and then sharpening the resulting signal. If no output texture is specified, the executable will launch an interactive viewer that supports iteratively stepping through the diffusion.
    Hit [SPACE] to start the iterative solver or hit "+" to advance one iteration at a time.
    --in <input mesh name>
    This string specifies the name of the mesh.
    [--inVF <vector-field file>]
    This string specifies the file containing the vector-field for visualization. (If this parameter is not specified, the principal curvature direction is used.)
    This file is assumed to be in binary, with the first four bytes storing an integer representing the number of vectors (this should be equal to the number of triangles in the mesh) followed by the list of vectors. The latter are encoded using double-precision floating point values and should be 8*num_triangles*dim bytes, with num_triangles the number of triangles/vectors and dim the dimension of vector-field. (The value of dim is equal to two if the --intrinsicVF is specified an three otherwise.)
    [--intrinsicVF]
    If enabled and a vector-field is specified, this flag indicates that the vector values are represented with two values per vector, using an intrinsic frame. Specifically, for triangle ( v0 , v1 , v2 ), the two-dimensional coefficients ( x , y ) correspond to the three-dimensional tangent vector ( x·(v1-v0) , y·(v2-v0) ).
    [--out <output texture>]
    This string is the name of the file to which the line-integral-convolution texture will be written.
    [--outVCycles <output v-cycles>]
    This integer specifies the number of v-cycles to use if the processed texture is output to a file and a direct solver is not used. The default value for this parameter is 10.
    [--licInterpolation <line-integral-convolution interpolation weight>]
    This floating point values gives the interpolation weight used for the line-integral-convolution.
    The default value for this parameter is 10000.
    [--sharpInterpolation <sharpening interpolation weight>]
    This floating point values gives the interpolation weight used for sharpening the line-integral-convolution results.
    The default value for this parameter is 10000.
    [--modulation <sharpening gradient modulation>]
    This floating point values gives the gradient modulation used for sharpening the line-integral-convolution results.
    The default value for this parameter is 100.
    [--width <output texture width>]
    This integers specifies the width of the output texture. The default value for this parameter is 2048.
    [--height <output texture height>]
    This integers specifies the height of the output texture. The default value for this parameter is 2048.
    [--jitter <random seed>]
    If specified, this integer value is used to seed the random number generation for jittering. (This is used to avoid singular situations when mesh vertices fall directly on edges in the texture grid. In such a situation, the executable will issue a warning "Zero row at index ...".)
    [--minor]
    If enabled, this flag specifies that the directions of minimal principal curvature should be used to define the vector-field (instead of the default maximal principal curvature directions).
    [--useDirectSolver]
    If enabled, this flag specifies that a direct solver should be used (instead of the default multigrid solver).
    Geodesics: An interactive tool for visualization of single-source geodesics using the heat method.
    In the interactive viewer the source can be set by holding the [SHIFT] key down and clicking/dragging with either mouse button.
    --in <input mesh name>
    This string specifies the the name of the mesh.
    [--interpolation <diffusion interpolation weight>]
    This floating point values gives the interpolation weight used for diffusing the initial delta function.
    The default value for this parameter is 10000.
    [--width <output texture width>]
    This integers specifies the width of the output texture. The default value for this parameter is 1024.
    [--height <output texture height>]
    This integers specifies the height of the output texture. The default value for this parameter is 1024.
    [--jitter <random seed>]
    If specified, this integer value is used to seed the random number generation for jittering. (This is used to avoid singular situations when mesh vertices fall directly on edges in the texture grid. In such a situation, the executable will issue a warning "Zero row at index ...".)
    [--useDirectSolver]
    If enabled, this flag specifies that a direct solver should be used (instead of the default multigrid solver).
    ReactionDiffusion: Performs simulation of reaction-diffusion based on the Gray-Scott model. If no output texture is specified, the executable will launch an interactive viewer that supports iteratively stepping through the reaction-diffusion process.
    Hit [SPACE] to start the reaction-diffusion process or hit "+" to advance one step at a time.
    --in <input mesh name>
    This string specifies the the name of the mesh.
    [--out <output texture>]
    This string is the name of the file to which the reaction-diffusion texture will be written.
    [--outSteps <output reaction-diffusion steps>]
    This integer specifies the number of reaction-diffusion steps to be taken. The default value for this parameter is 1000.
    [--width <output texture width>]
    This integers specifies the width of the output texture. The default value for this parameter is 512.
    [--height <output texture height>]
    This integers specifies the height of the output texture. The default value for this parameter is 512.
    [--jitter <random seed>]
    If specified, this integer value is used to seed the random number generation for jittering. (This is used to avoid singular situations when mesh vertices fall directly on edges in the texture grid. In such a situation, the executable will issue a warning "Zero row at index ...".)
    [--useDirectSolver]
    If enabled, this flag specifies that a direct solver should be used (instead of the default multigrid solver).
    [--dots]
    If enabled, this flag specifies that the feed/kill parameters for dot-formation should be used. Otherwise, the feed/kill parameters for stripes are used.
    TextureDilation: Dilates the texture by sampling across the chart seam and pulling interpolated texture values into the gutter texels.
    --in <input mesh and texture names>
    These two strings specify the the names of the mesh and the texture image.
    [--out <output texture mask>]
    This string is the name of the file to which the dilated texture will be written.
    [--radius <dilation radius>]
    This integer values gives the radius by which the texture should be dilated.
    The default value for this parameter is 0, indicating no dilation.
    [--verbose]
    If this flag is enabled, performance information is printed to stdout.
    TextureMasking: Identifies the active texels within a texture mask of prescribed resolution, with gutter texels rendered in red and texels supporting a triangle rendered in blue.
    --in <input mesh name>
    This string specifies the the name of the mesh.
    --res <texture width and texture height>
    These two integers specify the width and height of the mask.
    [--out <output texture mask>]
    This string is the name of the file to which the texture mask will be written.
    [--rasterizer]
    This integer specifies the type of information to be rasterized. Valid values are:
    • 0: active -- all texels whose support overlaps the texture atlas
    • 1: boundary -- all texels whose support overlaps the texture chart boundaries
    • 2: id -- colors texels by the triangle covering them
    • 3: node incidence count (unsigned) -- colors texels by the number of triangles sitting over them
    • 4: node incidence count (signed) -- colors texels by the number of positively oriented triangles over them, minus the number of negatively oriented triangles over them
    The default value for this parameter is 0 (active)
    SeamStitcher: Ensures an as-smooth-as-possible transition, either across chart boundaries or in regions expressly specified by an input mask, by replacing the values of texels with values minimizing the Dirichlet energy (subject to the constraint that the values of other texels remaine fixed).
    --in <input mesh and texture>
    These two strings specify the names of the mesh and the texture image.
    [--mask <input mask>]
    This string specifies the name of the mask image used to define texels whose value is to be replaced.
    Black pixels in the mask file indicate that the associated texels values should be updated to produce an as-smooth-as-possible image. (Results may be unpredictable if it is encoded using lossy compression.)
    If specified, the mask resolution should match that of the input texture.
    If not specified, values of texels at chart boundaries will be updated.
    [--out <output texture>]
    This string is the name of the file to which the smoothed texture will be written.
    [--exterior]
    If this flag is enabled (and a --mask parmeter is not specified), only the values of boundary texels whose centers are outside the charts are updated.
    [--verbose]
    If this flag is enabled, performance information is printed to stdout.
USAGE EXAMPLES (WITH SAMPLE DATA)
For testing purposes, a number of textured mapped models are provided (using the .ply extension). Of these, David and Julius include normal maps (using the .normap extension), Fertility includes the eight harmonic vector-fields (using the .vf extension), and Rooster uses (partial) texture maps as well as a mask image and confidence maps.
    TextureFiltering To run this executable you must specify the input mesh as well as the texture itself:
    % Bin/*/TextureFiltering --in ../TSP.Data/David/david.ply ../TSP.Data/David/david.normap
    This opens a viewer allowing the user to prescribe both global gradient modulation weights (through the slider) and local modulation weights (through a paint-brush interface, by depressing the [SHIFT] key and dragging with the left mouse button to smooth and the right mouse button to sharpen).
    You can also bypass the viewer and output a globally sharpened/smoothed texture to a file:
    % Bin/*/TextureFiltering --in ../TSP.Data/Julius/julius.ply ../TSP.Data/Julius/julius.normap --out julius.smooth.normap --modulation 0 --interpolation 100
    Here a modulation weight less than 1 indicates that gradients should be dampened (resulting in smoothing) and a small interpolation weight reduces the interpolation penalty, exaggerating the smoothing.
    TextureStitching This viewer can be run in one of two modes:
    1. In addition to the input mesh, specify a (single) composite texture and mask. If adjacent texels share the same mask color, they are assumed to come from the same source, and the gradient between them is preserved. Otherwise, the gradient is set to zero. Additionally, a mask color of black is reserved to indicate that the texel value is unknown.
      For example, running
      % Bin/*/TextureFiltering --in Rooster/rooster.ply ../TSP.Data/Rooster/texels.png --mask ../TSP.Data/Rooster/mask.png
      opens a viewer showing the stitched texture on the left and the composite texture on the right.
    2. In addition to the input mesh, specify (multiple) partial textures and associated confidence maps. The code blends the gradients in regions of overlap, with weights determined by the mask. Texel and confidence file names are specified using integer format specifiers, with zero-indexing. Colors are transformed to scalar confidence values by computing the gray-scale value and normalizing to the range [0,1].
      For example, running
      % Bin/*/TextureFiltering --in Rooster/rooster.ply ../TSP.Data/Rooster/texels-%02d.png --mask ../TSP.Data/Rooster/mask-%02d.png --multi
      opens a viewer showing the stitched texture on the left and the first partial textures on the right.
      Pressing the 't' key toggles forward through the partial textures and pressing 'T' toggles backwards.
      Holding [SHIFT] and clicking on the stitched model replaces the blended gradients under the paint-brush with the gradients from the currently visualized partial-texture.
    You can also bypass the viewer and output the stitched texture to a file:
    % Bin/*/TextureStitching --in Rooster/rooster.ply ../TSP.Data/Rooster/texels-%02d.png --mask ../TSP.Data/Rooster/mask-%02d.png --multi --out stitched.png
    LineIntegralConvolution To run this executable you must specify the input mesh:
    % Bin/*/LineIntegralConvolution --in ../TSP.Data/Fertility/fertility.ply
    This opens a viewer visualizing a vector-field by performing anisotropic diffusion to simulate line-integral-convolution. (To start the iterative solver, press the [SPACE] key.) By default, the vector-field used is defined by the (maximal) principal curvature directions.
    You can also explicitly prescribe the vector-field:
    % Bin/*/LineIntegralConvolution --in ../TSP.Data/Fertility/fertility.ply --inVF ../TSP.Data/Fertility/harmonic-001.vf --intrinsicVF
    (The --intrinsicVF flag is required because the vector-field in the file is represented using two intrinsic coordinates per triangle instead of three extrinsic ones.)
    You can also bypass the viewer and output the line-integral-convolution texture to a file:
    % Bin/*/LineIntegralConvolution --in ../TSP.Data/Hand/hand.ply --minimal --out hand.minimal.jpg
    Here a visualization of the minimal principal curvature directions is written out as a texture image.
    Geodesics To run this executable you must specify the input mesh:
    % Bin/*/Geodesics --in ../TSP.Data/Bunny/bunny.ply
    This opens a viewer allowing the user to prescribe the source of the geodesic by holding the [SHIFT] button and clicking on the source location with either mouse button.
    ReactionDiffusion To run this executable you must specify the input mesh:
    % Bin/*/ReactionDiffusion --in ../TSP.Data/Camel/camel.ply
    This opens a viewer visualizing the "stripes" reaction-diffusion process. (To start the process, press the [SPACE] key.)
    You can also bypass the viewer and output the reaction-diffusion texture to a file:
    % Bin/*/ReactionDiffusion --in ../TSP.Data/David/david.ply --out david.dots.jpg --dots --outSteps 2000
    Here a "dots" pattern is written out to an image. (Empirically, we have found that this reaction-diffusion process takes more steps to converge, hence the larger number of steps.)
HEADER-ONLY LIBRARY
    include/Src/GradientDomain.h In addition to executables, the gradient-domain processing code can be interfaced through either the GradientDomain class or the HierarchicalGradientDomain class declared in include/Src/GradientDomain.h.
      GradientDomain: Once constructed, this object can be used to query the active texels/edges, as well as create the standard mass/stiffness/divergence matrices and apply the mass/stiffness/operators to the texel data. In the descriptions below, the template parameter Real is the floating point type used to represent data (typically double) and Solver is the class used to factor and solve the sparse system of linear equations (Eigen::SimplicialLDLT by default). Code Description:
        The code performs basic gradient domain processing applications including texture smoothing/sharpening and stitching. This is done by solving for the output texture values which simultaneouly fit value and derivative constraints.
        • Value constraints are described by specifying the desired values at the texels, defined to be the input texel values.
        • Gradient constraints are described by specifying the desired differences across edges between texels, defined to be the dampened/amplified differences between input texture values (and zerod out if the texels come from different patches, in the case of stitching).

      Code walk-through:

        The details of the implementation can be found in the GradientDomain.example.cpp code.
        • Lines 113-125: The texture-mapped geometry and the texture image (as well as a mask image describing when texels belong to the same patch, for stitching) are read in.
        • Lines 143-157: The GradientDomain object is constructed, passing in the resolution of the mesh as well as functor giving the indices of embedding/texture-vertices for each corner, functors giving the positions of embedding/texture-vertices, the texture image resolution, and the number of quadrature points per triangle used for integration (valid values are 1, 3, 6, 12, 24, and 32).
        • Lines 161-166: The input texture values are read from the image into a std::vector, using the member functions GradientDomain::numNodes to get the number of (active) texels in the texture map and GradientDomain::node to get the coordinates of the texel within the image.
        • Lines 168-195: The constraints to the linear system are constructed, specifying the target values and gradients:
          • Lines 172-173: The target value constraints are constructed by applying the mass matrix to the input texel values.
          • Lines 175-190: The target gradient constraints are obtained by computing the target per-edge differences and then computing the divergence:
            • Lines 177-186: The target edge differences are obtained by iterating over the edges, computing the difference between the input texel values at the end-points, and scaling by the gradient modulation value (and zeroing out the difference in the case the end-points are assigned different IDs, in the case of stiching). To this end, the member function GradientDomain::numEdges gives the number of edges, and the member function GradientDomain::edge returns the indices of the edge's two end-points.
            • Lines 188-189: The target gradient constraints are obtained applying the divergence operator to the computed edge differences.
          • Lines 192-193: The target value and gradient weights are combined using the weights specified by the user.
        • Lines 197-199: The system matrix is constructed by taking the weighted combination of the mass and stiffness matrices (using the same weights for combining the value and gradient constraints).
        • Lines 201-211: The system matrix is factored.
        • Lines 213-221: The values for the individual image channels are computed by solving the linear system.
        • Lines 223-228: The output texel values are written from the std::vector back into the texture image.

      Assumptions:

        The code make a number of assumptions about the input geometry:
        • The code should support non-injective texture mappings.
        • For numerical purposes neither surface nor texture triangles should be degenerate.
        • The indexing of surface vertices is such that the topology implied by the vertex indexing matches that of the surface.
        • The indexing of texture vertices is such that the topology implied by the vertex indexing matches the toplogy of the texture atlas. (i.e. A single surface vertex can be associated with different texture vertices if the associated corners are in different charts.)
      HierarchicalGradientDomain: This class derives from GradientDomain and additionally supports an iterative multigrid solver for computing the solution to the gradient domain problem without requiring a potentially expensive matrix factorization. The interface is simlar to that of GradientDomain and the details of the implementation can be found in HierarchicalGradientDomain.example.cpp. What follows is a list of the minor changes needed in order to access and use the hierarchical solver:
      • Line 152-167: The interface uses the class HierarchicalGradientDomain, whose constructor takes an additional parameter describing the number of levels in the multigrid hierarchy.
        Note: Because the class is also responsible for relaxing the gradient-domain system, it is templated off the type of the direct solver used to solve the system at the coarsest resolution (second argument) and the type of data being solved for (third argument).
        [Compare to Lines 143-157 of GradientDomainExample.cpp]
      • Line 170: The HierarchicalGradientDomain class maintains its own constraint and solution vectors. So rather than allocating the constraint and solution vectors separately, the interface uses pointers to the data maintained in the hgd object.
        [Compare to Line 159 of GradientDomainExample.cpp]
      • Lines 208-210: The HierarchicalGradientDomain class maintains its own representation of the system matrix. This is initialized by specifying the mass and stiffness weights for the system.
        [Compare to Lines 197-199 of GradientDomainExample.cpp]
      • Line 212-214: The solution is relaxed by running the prescribed number of v-cycles with the prescribed number of Gauss-Seidel relaxations per hierarchy level.
        [Compare to Lines 201-221 of GradientDomainExample.cpp]
COMPILATION AND EXECUTION
  • The Windows executables require both the glew and glut dynamically linked libraries to run. These can be found here and should be included either in the directory with the executables, or in the directory from which the executables are run.
  • Compiling under Windows requires both the glew and glut libraries. These can be found here and should be placed in the output directory for linkage.
  • Compilation requires a linear solver. By default, we use the LDLt implementation provided by Eigen. If you have Intel's oneMKL, we encourage you to use Eigen's Pardiso implementation. To to this you will need to enable the USE_EIGEN_PARDISO flag in include/Src/PreProcessing.h
HISTORY OF CHANGES

Version 2.00:

  • Added support for reaction-diffusion based on the Gray-Scott model.

Version 3.00:

  • Added support for texture stitching.

Version 4.00:

  • Added Makefile.no_visual to allow building texture filtering/stitching applications without visualizations.

Version 4.01:

  • Added support for reading .obj files.

Version 4.02:

  • Added support for mask visualization.
  • Switched exceptions to warnings.

Version 4.03:

  • Added support for segment boundary dilation in the TextureStitching code.

Version 4.05:

  • Modified the --jitter flag to take a random seed.

Version 4.06:

  • Added support for visualizing weights when using multi-stitching.

Version 4.07:

  • Added support for providing a separate low-frequency signal for texture processing.

Version 4.08:

  • Removing numerical issues in loop construction.

Version 4.50:

  • Code clean-up

Version 4.60:

  • Added --mask for specifying mask(s) to support default cross-chart smoothing.

Version 4.75:

  • Added support for 16-bit png files.

Version 5.00:

  • Separated out OpenMP depeendency.

Version 5.01:

  • Add support for processing polygonal (i.e. not necessarily triangular) faces.

Version 6.00:

  • Add funcionality for dilating the texture map and masking out active texels.

Version 6.05:

  • Fixed off-by-half-pixel issue with texture rasterization.

Version 6.06:

  • Cleaned up texture dilation
  • Made non-template, header-only functions inline.

Version 7.00:

  • Modified to support non-bijectve texture maps.
  • Removed dependence on Triangle code.

Version 8.00:

  • Added header-only library for standard geometry-prcoessing interfaces, wrapped in include/Src/GradientDomain.h
  • Added example code showing how to use the libary in GradientDomain.example.cpp.

Version 9.00:

  • Added SeamStitcher executable.

Version 9.05:

  • Added the option to provide a mask to the SeamStitcher executable to indicate which texels are to be locked.

Version 9.10:

  • Added options for TextureMasking.
  • Added more numerical stability.

Version 9.10:

  • Added header-only library for a multigrid solver supporting standard geometry-prcoessing interfaces, wrapped in include/Src/GradientDomain.h
  • Added example code showing how to use the libary in HierarchicalGradientDomain.example.cpp.
SUPPORT
This work genersouly supported by NSF grant #1422325.
HOME

About

Gradient-Domain Processing within a Texture Atlas

Resources

Readme

License

MIT license
Activity

Stars

76 stars

Watchers

7 watching

Forks

12 forks
Report repository

Releases

No releases published

Packages

No packages published

Contributors 2

  •  
  •  

Languages