User Tools

Site Tools


guides:plotting:plotmanlinux

Automated plotting on Linux with Plotman

Reference Sites

  1. Install on plotting machine
  2. Configure archiver to move plots to external or remote drives

Installing Plotman

  1. Ensure that you have at least Python 3.7 installed and Pip installed
  2. Activate your Chia environment
    • . ./activate
  3. Install Plotman
  4. Verify that Plotman is working with
    • plotman status

Configure Plotting and Destination Drives

  1. Create a default plotman.yaml config file
    • plotman config generate
    • This will be located in ~/.config/plotman/plotman.yaml
  2. Plotman is designed to maximize the number of plots running in parallel
    • You need to tell it which drives you use for
      • plotting (primary and secondary)
      • destination
      • (optionally) archiving
    • And how to stagger jobs and the amount of resources to use
    • An example config for plotting with two SSD's, a second plotting temp drive, and a single destination drive
      • directories:
           # normally NVMe SSD or SAS
           tmp:
              - /data/s970evop_1
              - /data/mp510_1
        
           # only worthwhile if NVMe or SATA SSD
           tmp2: /data/crp1_1/tmp
        
           # normally spinning rust
           dst:
              - /data/wdredp_1

Configure Job Staggering

  • An example config that allows 6 parallel jobs in total, with no more than 3 jobs per primary plotting drive, but only 2 jobs per drive in phase 1
    • scheduling:
         tmpdir_stagger_phase_major: 2
         tmpdir_stagger_phase_minor: 1
         tmpdir_stagger_phase_limit: 2   # no more than 2 jobs per drive for above phase thresholds
         tmpdir_max_jobs: 3
         global_max_jobs: 6

Configure Plotman Plotting for Pools

  • To allow creation of plots for a pool, add the pool contract address to the config file
    • plotting:
         pool_contract_address: xch..............
    • Find your pool contract address with the command
      • chia plotnft show

Configure Chia Plot Parameters

  • Example for rig with 12 hyperthreads, 32 GB RAM, and max of 6 parallel jobs
    • plotting:
      type: chia
      chia:
        k: 32                # k-size of plot, leave at 32 most of the time
        e: False             # Use -e plotting option
        n_threads: 4         # Threads per job: Can overcommit providing no more than 4 jobs allowed in phase 1
        n_buckets: 128       # Number of buckets to split data into
        job_buffer: 5120     # Per job memory: Increase above default 3390 if RAM to spare for marginal improvement

Configure Archiving

  • Plotman allows optional archiving to either local or remote drives
    • Although rsyncd is recommended for remote drives, it is also possible to use remote NFS drives and Plotman will think that they are local drives
    • An example config for archiving with throttled bandwidth to a harvester running on a Raspberry Pi with NFS drives is
      • archiving:
          target: rsync_rsync_throttle
          env:
            site_root: /nfs/chia
          target_definitions:
            rsync_rsync_throttle:
              env:
                command: rsync
                # adjust bandwidth limit in bytes/second to avoid overloading device hosting archive drives
                options: --bwlimit=35000 --preallocate --remove-source-files --skip-compress plot --whole-file
                site_root: null
              disk_space_script: |
                #!/bin/bash
                set -evx
                df -BK | grep " ${site_root}/" | awk '{ gsub(/K$/,"",$4); printf "%s:%.0f\n", $6, $4*1024 }'
              transfer_script: |
                #!/bin/bash
                set -evx
                "${command}" ${options} "${source}" "${destination}"
              transfer_process_name: "{command}"
              transfer_process_argument_prefix: "{site_root}"

Running Plotman

  • From the CLI the three commands you will use the most are
    • plotman status
      • Gives an overview of all the jobs running
    • plotman interactive
      • Interactive screen showing jobs and archiving progress
      • Works well with headless systems, i.e. SSH friendly
      • Stop and start plotting and archiving using the <p> and <a> keys
      • Quit the interactive screen by pressing <q>
    • plotman analyze
      • Allows you to see how many jobs you have created over period
      • For example:
      • plotman analyze ~/.chia/mainnet/log/*2021-08-09*
      • +-------+----+-------------+---------------+--------------+---------------+---------------+--------------+
        | Slice | n  |   %usort    |    phase 1    |   phase 2    |    phase 3    |    phase 4    |  total time  |
        +=======+====+=============+===============+==============+===============+===============+==============+
        | x     | 18 | μ=100.0 σ=0 | μ=12.6K σ=659 | μ=5.4K σ=307 | μ=11.6K σ=817 | μ=854.6 σ=132 | μ=30.5K σ=1K |
        +-------+----+-------------+---------------+--------------+---------------+---------------+--------------+
guides/plotting/plotmanlinux.txt · Last modified: 2021/08/26 21:24 by spacefarmer01