Sync server

The Sync Server is a Go binary installed on an Alpine Linux Docker image that polls the wp.org API's for themes and plugins to process and writes them to a queue.

Unless you have good reason to set the SYNC_ACTIVE environment variable to on you should leave the server set to off and not bring up the container by running make sync.up. Turning on the Sync Server will put around 50k audits in the queue and will take days to process locally and potentially lock up all your computers resources. A better way to audit wp.org themes and plugins would be to follow the instructions found in the Run Audits section of the setup page.

If you do decide that you would like to seed the queue with plugins and themes, or are testing changes you've made to the Go binary, you can run make sync.up for a second or two and then in a different shell run make sync.down which should add a few hundred items to the queue.

Environment Variables

Variable Description
SYNC_ACTIVE Whether the Sync Server is active or not. Must be one of: on, off. Default is off.
SYNC_API_BROWSE_CATEGORY The API category used to ingest the wp.org themes and plugins. Must be one of: popular, featured, updated, new. Default is updated.
SYNC_DATA When the database provider is set to local this will be where the data is stored relative to the /srv/data working directory. Default is ./db.
SYNC_DATABASE_DOCUMENT_PATH When the database provider is set to firestore this value is the path to the document in Cloud Firestore. Must be in the form of <collection>/<document>. Default is sync-server/wporg.
SYNC_DATABASE_PROVIDER Tells the Sync Server which database provider to use; either the local file system or Google Cloud Firestore. Must be one of: local, firestore. Default is local.
SYNC_DEFAULT_CLIENT The API client used to make requests by the audit servers; also associated with the key and secret those server use. Default is wporg.
SYNC_DEFAULT_VISIBILITY The audit and report visibility. Must be one of: public, private. Default is public.
SYNC_FORCE_AUDITS Forces audit reports to be generated even if a report exists for the checksum and standard. Must be one of: yes, no. Default is no.
SYNC_ITEMS_PER_PAGE The number of plugins or themes per page in the API request. Default is 250.
SYNC_LH_ACTIVE Send messages to the Lighthouse SQS queue. Must be one of: on, off. Default is on.
SYNC_MESSAGE_PROVIDER Queue audit messages using the local MongoDB, Google Cloud Firestore, or AWS SQS. Must be one of: local, firestore, sqs. Default is local.
SYNC_PHPCS_ACTIVE Send messages to the PHPCS SQS queue. Must be one of: on, off. Default is on.
SYNC_POOL_DELAY The wait time in seconds between the wp.org theme and plugin ingests. Default is 600.
SYNC_POOL_WORKERS The number of workers (concurrent goroutines) the server will create to ingest the wp.org API. Default is 125.

Commands

Command Description
make sync.build.bin Build the Sync Server Go binary.
make sync.clean.bin Clean the Sync Server Go binary.
make sync.build.image Build the Sync Server Docker image.
make sync.build.up Rebuild the Sync Server Docker image & run the container in isolation with docker-compose up.
make sync.up Run the Sync Server Docker container in isolation with docker-compose up.
make sync.down Take the isolated Sync Server Docker container down.
make sync.stop Stop the isolated Sync Server Docker container with docker-compose stop.
make sync.rm Remove the stopped Sync Server Docker container with docker-compose rm.
make sync.push.image Push the Sync Server Docker image to GCR.
make sync.clean.image Clean the Sync Server Docker image from the host machine.
make sync.build.disk Build the Sync Server GKE persistent disk.
make sync.build.cluster Build the Sync Server GKE cluster.
make sync.creds Get the Sync Server GKE cluster credentials.
make sync.tpl Generate the Sync Server GKE YAML template.
make sync.deploy.cluster Deploy the Sync Server GKE cluster.
make sync.get.cluster Get the Sync Server GKE cluster status.
make sync.clean.cluster Clean the Sync Server GKE cluster.